Line of Codes

Now the essential part of MIB Parser is completed and achieved its objective.

Here is LoC required to get it to current state.

• Xtext (134 lines)
• Xcore ( 107 lines)
• Java (6 lines)
• Xtend (27 lines)
• Acceleo (31 lines)

Some of the xtend codes that I written earlier for scoping is not counted as its deemed as not required once I understand namespace concept in Xtext.

Generate Oid list from Mib model

To generate Oid list (i.e text file) from Mib model, there are a few readily available Model to Text Transformation (M2T) tools for EMF.

• Acceleo
• Xtend
• Jet

I have started this work on Acceleo, and find the editor's content assist is pretty good and and straight forward way to apply launcher menu for mib files - almost no coding except the template declaration which is based on MOFM2T.

I will try to detail the steps clearly as I could for others to benefit. I struggled to get these end to end steps correct as most of tutorials out there are by features and less based on use-case (on top of that I used Xcore for model, hence it was a blocker at start).

1. Install Acceleo from Modelling Component place (Version: 3.5.1), and change to Acceleo perspective, this makes lots of handy tool/wizard simple to find as oppose continue working on Java perspective.
2. Acceleo Create Project wizard does not recognize Xcore as model. Therefore, I had to generate Ecore model first. Choose Mib.xcore from Package Explorer, and click "new -> Other..", choose EMF Generator Model. This will generate both Ecore and genmodel.
3. Create a new Acceleo project "com.ravi.mib.acceleo.module.oid". On second wizard page - "Create a new Acceleo generation file", change module name to "oid" and search for Mib model from "Runtime Version" (http://www.ravi.com/ibm/xtext/Mib). Change template name to "generateOid" and type to "MibModel"
4. Once the project created, edit com/ravi/mib/acceleo/module/oid/common/oid.mtl file and apply following template.

oid.mtl

[comment encoding = UTF-8 /]
[module oid('http://www.ravi.com/mib/xtext/Mib')]

[template public generateOID(model : MibModel)]
 [comment @main /]
 [for (d : Definition | model.definitions)]
  [file (d.name.toString().concat('.oid'), false, 'UTF-8')]
   [for (oid : Identifier | d.identifiers)]
    [if (isValidObjectType(oid))]
[oid.name/],[oid.value.oid/],[getPrimitive(oid)/],[getDerived(oid)/]
    [/if]
   [/for]
  [/file]
 [/for]
[/template]

[query private isValidObjectType(oid : Identifier) : Boolean = 
 oid.oclIsTypeOf(ObjectType) and
 not oid.oclAsType(ObjectType).type.typeName.oclIsUndefined() and
 oid.oclAsType(ObjectType).type.typeName.toString().strcmp('OBJECT IDENTIFIER') <> 0
/]

[query private getPrimitive(oid : Identifier) : String =
  if oid.oclAsType(ObjectType).type.dataType.derived.name.oclIsUndefined() then
  ''
 else
  oid.oclAsType(ObjectType).type.dataType.derived.name.toString()
 endif
/]

[query private getDerived(oid : Identifier) : String = 
 oid.oclAsType(ObjectType).type.typeName.toString()
/]



5. There will be an error afterwards for "oid.common" package to be exported. Do as suggested.

6. Now for launcher menu, select earlier created Acceleo project from Package Explorer and choose new Acceleo UI Launcher project, On "Acceleo UI Launcher settings", change model file name filter to "*.mib" and target folder access to "...getFolder("oid")".
7. Finally modify label for menu as "Mib Generator"

8. Once eclipse instance is launched, Generate Oid option should available for mib files.

RFC1213-MIB.oid

sysDescr,1.3.6.1.2.1.1.1,DisplayString,OCTET STRING
sysUpTime,1.3.6.1.2.1.1.3,TimeTicks,INTEGER
sysContact,1.3.6.1.2.1.1.4,DisplayString,OCTET STRING
sysName,1.3.6.1.2.1.1.5,DisplayString,OCTET STRING
sysLocation,1.3.6.1.2.1.1.6,DisplayString,OCTET STRING
sysServices,1.3.6.1.2.1.1.7,,INTEGER
ifNumber,1.3.6.1.2.1.2.1,,INTEGER
ifIndex,1.3.6.1.2.1.2.2.1.1,,INTEGER
ifDescr,1.3.6.1.2.1.2.2.1.2,DisplayString,OCTET STRING
ifType,1.3.6.1.2.1.2.2.1.3,,INTEGER
ifMtu,1.3.6.1.2.1.2.2.1.4,,INTEGER
ifSpeed,1.3.6.1.2.1.2.2.1.5,Gauge,INTEGER
ifPhysAddress,1.3.6.1.2.1.2.2.1.6,PhysAddress,OCTET STRING
ifAdminStatus,1.3.6.1.2.1.2.2.1.7,,INTEGER
ifOperStatus,1.3.6.1.2.1.2.2.1.8,,INTEGER
ifLastChange,1.3.6.1.2.1.2.2.1.9,TimeTicks,INTEGER
ifInOctets,1.3.6.1.2.1.2.2.1.10,Counter,INTEGER
ifInUcastPkts,1.3.6.1.2.1.2.2.1.11,Counter,INTEGER
ifInNUcastPkts,1.3.6.1.2.1.2.2.1.12,Counter,INTEGER
ifInDiscards,1.3.6.1.2.1.2.2.1.13,Counter,INTEGER
ifInErrors,1.3.6.1.2.1.2.2.1.14,Counter,INTEGER
ifInUnknownProtos,1.3.6.1.2.1.2.2.1.15,Counter,INTEGER
ifOutOctets,1.3.6.1.2.1.2.2.1.16,Counter,INTEGER
ifOutUcastPkts,1.3.6.1.2.1.2.2.1.17,Counter,INTEGER
ifOutNUcastPkts,1.3.6.1.2.1.2.2.1.18,Counter,INTEGER
ifOutDiscards,1.3.6.1.2.1.2.2.1.19,Counter,INTEGER
ifOutErrors,1.3.6.1.2.1.2.2.1.20,Counter,INTEGER
ifOutQLen,1.3.6.1.2.1.2.2.1.21,Gauge,INTEGER
atIfIndex,1.3.6.1.2.1.3.1.1.1,,INTEGER

TODO: Is there any way for Xcore to register its model URI so that Acceleo able to recognize without going through Ecore generation?

TODO: How to limit OID list down to the selected mib file alone?

Completing Data Type reference in Xtext Grammar

As described in overview from the first post, its almost close to produce CSV artefacts containing OID short name, Oid in dot notation number, derived data type and its primitive data type.

The part missing from model now is the data type information. This can be completed with following modification to Xtext grammar and Xcore.

Mib.xtext

ObjectType:
 name=ID imp=[Macro] 
 'SYNTAX' type=mibType2
 'ACCESS' access=AccessEnum 
 'STATUS' status=StatusEnum 
 ('DESCRIPTION' description=STRING)? 
 ('REFERENCE' reference=STRING)? 
 ('INDEX' '{' mibType (',' mibType)* '}')? 
 ('DEFVAL' '{' INT | STRING '}')? 
 value=OidValue;

ObjectType class will now contain feature type of mibType2 among others. mibType2 is similiar to mibType, the reason I have created another instance of mibType is as it used in lot of places such as Macro, Object Type Index, Choice etc. All these rightfully need to be changed as well, however for now I want to keep it simple - it can be overwhelming otherwise.

Mib.xtext

mibType2:
 (sequence?=SequenceOf)? 
 dataType=DataType
 ('(' 'SIZE' '(' (INT '..')? INT ')'')')? 
 ('(' (INT '..')? (INT | 'MAX') ')')? 
 ('{' ID '(' INT ')' (',' ID '(' INT ')')* '}')?;

DataType:
 name=(OctectString | 'INTEGER' | ObjectIdentifier) |
 derived=[TypeDefinition];

SequenceOf:
 'SEQUENCE' 'OF';
 
ObjectIdentifier:
 'OBJECT' 'IDENTIFIER';

OctectString:
 'OCTET' 'STRING';

One trick I learned here is that if there are multiple keywords make up a syntax, its better to push those down to its own rule e.g SequenceOf - Xtext  intelligent enough to know this is terminal like rule and will not create an EClass for it.

Datatype takes primitive data types (3 types) or a derived data type from TypeDefinition.  Note that this 'TypeDefinition' used to be called 'DataType' earlier - re-factored to reflect semantic better.

Mib.xtext

Definition:
 name=ID 'DEFINITIONS' '::=' 'BEGIN'
 Export?
 imports=Import?
 (identifiers+=Identifier | typedef+=TypeDefinition | macros+=Macro)+
 'END';

TypeDefinition:
 name=ID '::=' ('[' 'APPLICATION' INT ']')? 
 'IMPLICIT'? (Choice | Sequence | type=mibType2);

And to reflect this on Xcore, following need to be changed:

Mib.xtext

class Definition {
 String name
 contains Import imports
 contains Identifier[] identifiers
 contains TypeDefinition[] typedef
 contains Macro[] macros
}

class TypeDefinition {
 String name
 contains mibType2 ^type
}

Setting TypeDefinition as containment for Definition allows Object Type's type to refer to this.

Mib.xtext

class ObjectType extends Identifier {
 refers Macro imp
 contains mibType2 ^type
 AccessEnum access
 StatusEnum status
 String description
 String reference
}

enum AccessEnum {
 readOnly as "read-only"
 readWrite as "read-write" = 1
 writeOnly as "write-only" = 2
 notAccessible as "not-accessible" = 3
}

enum StatusEnum {
 mandatory
 optional = 1
 obsolete = 2
 deprecated = 3
}

Now running this will give full "open declaration" for type with derived data type as well.

Generate dot notation OIDs - Part 2

Following last post, there are few corrections are required both in term of accuracy and correctness.

1. Transmission OID
I referred Object Identifier transmission as 0.3.6.1.2.1.10, this clearly wrong as the correct reference is "1.3.6.1.2.1.10". This is due to dependency module RFC1155-SMI from RFC 1155 does not provide an oidnum for "iso".


RFC1155-SMI.txt

        -- Next original line is commented out for simpler syntax
 -- internet      OBJECT IDENTIFIER ::= { iso org(3) dod(6) 1 }
 iso           OBJECT IDENTIFIER ::= { 0 }
 org           OBJECT IDENTIFIER ::= { iso 3 }
 dod           OBJECT IDENTIFIER ::= { org 6 }
 internet      OBJECT IDENTIFIER ::= { dod 1 }


Earlier I have mistakenly associate iso as '0' hence the error. This should fix it.


RFC1155-SMI.txt

 iso           OBJECT IDENTIFIER ::= { 1 }

Now this produces following output:



2. MIB terminology
Referring dot notation oid string as "Dotted Oid" is not accurate as oid generally refers to dot notation itself, hence again I modified the feature name to just refer it as "Oid" (refer to image above)

Mib.xcore

class OidValue {
 refers Identifier parent
 int oidnum
 derived String oid get {
  var p = parent
  var doid = oidnum.toString
  while (p != null) {
   doid = p.value.oidnum + "." + doid
   p = p.value.parent
  }
  doid
 }
}

Another thing worth to mention here is the usage of Xcore. Xcore is an alternative to common Ecore editor with additional behaviour definitions (class method, derived feature).

TODO: However in above example "derived String oid get {..}", my initial impression was oid is lazy initialization that is set when getOid() is called, however this is not the case - from the generated code, the oid feature does not exist at all, its purely a method that does the calculation every time its called. Therefore the advantage of this "derived String oid get {..}" vs "op String getOid {..}" is the former available within the class feature.

OidValueImpl.java

 @Override
 public Object eGet(int featureID, boolean resolve, boolean coreType) {
  switch (featureID) {
   case MibPackage.OID_VALUE__PARENT:
    if (resolve) return getParent();
    return basicGetParent();
   case MibPackage.OID_VALUE__OIDNUM:
    return getOidnum();
   case MibPackage.OID_VALUE__OID:
    return getOid();
  }
  return super.eGet(featureID, resolve, coreType);
 }

Hope there is a way to make it work as lazy initialization. However, understandably lazy initialization is not going to be an easy implementation, as notification mechanism should be considered as well i.e any change to referent object should re-trigger the evaluation.

TODO2:  How does this set works i.e "derived String oid set {..}"? how do we access the parameters?

Generate dot notation OIDs - Part 1

Now basic MIB file parsing is working, its always important to get full dotted name and oid.

In order to do this, there are 2 different approaches:

• Add a feature within Identifier EObject that calculates the dotted name/oid and saves the value as a feature (attribute).
• Do Model to Model Transformation (MMT).

I will opt for the first as this is rather a natural addition to the existing MIB model. Came across XCore that does provide exactly this and the functionality is called "Derived Feature".

To enable this, need to:

• Take over the Ecore/Genmodel generation from Xtext 
• Export this Ecore/Genmodel to Xcore
• Finally make Xtext to refer to the Xcore instead.

Here are the actual steps:

• Make sure Xcore is available in Eclipse

• Before exporting Mib.genmodel to Xcore, make necessary changes for generation directories/id. Because once exported to Xcore, all the files are auto-generated - its messy work to clean up later.

• Export Mib.genmodel to Xcore. Specify the location to "platform:/resource/com.ravi.mib.xtext/model/xcore/". Noticed it always fail on the first attempt and successful on the second.

• Import Mib model instead of generating it:

Mib.xtext

//generate mib "http://www.ravi.com/mib/xtext/Mib"
import "http://www.ravi.com/mib/xtext/Mib"

• Modify MWE to stop generating ECore and re-point the Xcore:

GenerateMib.mwe2

    component = Generator {
        pathRtProject = runtimeProject
        pathUiProject = "${runtimeProject}.ui"
        pathTestProject = "${runtimeProject}.tests"
        projectNameRt = projectName
        projectNameUi = "${projectName}.ui"
        encoding = encoding
        language = auto-inject {
            // switch to xcore
            loadedResource = "platform:/resource/org.eclipse.emf.ecore.xcore.lib/model/XcoreLang.xcore"
            loadedResource = "classpath:/model/Xbase.ecore"
            loadedResource = "classpath:/model/Xbase.genmodel"
            loadedResource = "classpath:/model/Ecore.ecore"
            loadedResource = "classpath:/model/Ecore.genmodel"
            loadedResource = "platform:/resource/${projectName}/model/xcore/Mib.xcore"
            uri = grammarURI
    
            // Java API to access grammar elements (required by several other fragments)
            fragment = grammarAccess.GrammarAccessFragment auto-inject {}
    
            // generates Java API for the generated EPackages
            // switch to xcore
            // fragment = ecore.EMFGeneratorFragment auto-inject {}
    

• Add Xcore plugin to MANIFEST.MF:

MANIFEST.MF

Require-Bundle: org.eclipse.xtext;visibility:=reexport,
 ...
 org.eclipse.xtext.xbase.lib,
 org.eclipse.emf.ecore.xcore
Import-Package: org.apache.log4j,

• Now, running MWE should not cause any error (besides maybe need to manual trigger Xcore code generation - by making the Xcore file dirty e.g enter space and save)

TODO: How to add Xcore generation in MWE?

Now add following derive feature to Xcore to generate dotted name (under Identifier) and dotted oid (under OidValue).

Mib.xcore

class Identifier {
 String name
 contains OidValue value
 derived String dottedName get {
  var p = value.parent
  var dname = name
  while (p != null) {
   dname = p.name + "." + dname
   p = p.value.parent
  }
  dname
 }
}

...
class OidValue {
 refers Identifier parent
 int oidnum
 derived String dottedOid get {
  var p = parent
  var doid = oidnum.toString
  while (p != null) {
   doid = p.value.oidnum + "." + doid
   p = p.value.parent
  }
  doid
 }
}

Now, dotted name and oid should be available in the MIB model (open this via Mib Model Editor instead of Xtext Mib editor).

Import objects should link to proper source MIB definition - Part 3

From last Mib.xtext, the main change need to be done is to break Mib Objects from Import Objects. i.e The Mib Object identifier does not need to directly point to import section's identifier. Import works more as 'namespace' lookup for each identifier. Therefore Mib grammar should be modified as follow:

Mib.xtext

Import:
 name='IMPORTS' defs+=ImpDef+ ';';

ImpDef:
 objects+=ImpObject (',' objects+=ImpObject)* 'FROM' name=ID;

ImpObject:
 name=ID;

And re-point imp to macro as well as simplify Identifier:

Mib.xtext

Identifier:
 ObjectType | (name=ID mibType value=OidValue);
// ObjectType | (name=MibObject mibType value=OidValue);

ObjectType:
 name=ID imp=[Macro] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
// name=MibObject imp=[MibObject] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
 ('DESCRIPTION' STRING)?
 ('REFERENCE' STRING)?
 ('INDEX' '{' mibType (',' mibType)* '}')?
 ('DEFVAL' '{' INT | STRING '}')?
 value=OidValue;

OidValue:
 '::=' '{' parent=[Identifier]? oidnum=INT '}';
// '::=' '{' parent=[MibObject]? oidnum=INT '}';

And since we are not using "importedNamespace" attribute, we need to complete the namespace by overriding "internalGetImportedNamespaceResolvers()" - note that I used "getImportedNamespace()" on earlier example - but that can't be used here as Xtext not able to know all the imported object since the structure is pretty complex in MIB. Hence overriding the "internalGetImportedNamespaceResolvers()" which is in fact the caller for "getImportedNamespace()" is more appropriate here.

MibImportedNamespaceAwareLocalScopeProvider.xtend

class MibImportedNamespaceAwareLocalScopeProvider extends ImportedNamespaceAwareLocalScopeProvider {
 override List internalGetImportedNamespaceResolvers(EObject context, boolean ignoreCase) {
  val list = new LinkedList
  if (context instanceof Definition) {
   if ((context as Definition).imports != null) {
    for (x : EcoreUtil2.getAllContentsOfType((context as Definition).imports, ImpObject)) {
     val qname = (x.eContainer as ImpDef).name + "." + x.name
     list.add(createImportedNamespaceResolver(qname, ignoreCase));
    }
   }
  } 
  list
 }
}

And of course bind this class by:

MibRuntimeModule.java

public class MibRuntimeModule extends
  com.ravi.mib.xtext.AbstractMibRuntimeModule {
 @Override
 public void configureIScopeProviderDelegate(com.google.inject.Binder binder) {
  binder.bind(org.eclipse.xtext.scoping.IScopeProvider.class)
    .annotatedWith(
      com.google.inject.name.Names
        .named(org.eclipse.xtext.scoping.impl.AbstractDeclarativeScopeProvider.NAMED_DELEGATE))
    .to(MibImportedNamespaceAwareLocalScopeProvider.class);
 }
}

Now, "Open-Declaration" (i.e F3) feature should work for both "Mib object identifier" and "OBJECT-TYPE" macro, open up file that contains those definitions.

Import objects should link to proper source MIB definition - Part 2

Now will apply the same changes to Mib project.

As preparation, need to get all dependent MIB files:

• RFC1155-SMI.mib (Extract from rfc, section 6 - Definitions)
• RFC-1212.mib (Extract from rfc, section 4 - Defining Objects, need to complete it manually)
• RFC1158-MIB.mib (Extract from rfc, section 6 - Definitions) - This is dependent for RFC-1212 on "DisplayString" definition only. This is obsoleted by RFC1213.

Open each files in Xtext Editor, make sure MIB grammar able to handle them.

 

RFC1155-SMI.mib

Following format is not supported:

RFC1155-SMI.mib

internet      OBJECT IDENTIFIER ::= { iso org(3) dod(6) 1 }

Therefore, modify the file to following

RFC1155-SMI.mib

-- internet      OBJECT IDENTIFIER ::= { iso org(3) dod(6) 1 }

iso           OBJECT IDENTIFIER ::= { 0 }

org           OBJECT IDENTIFIER ::= { iso 3 }

dod           OBJECT IDENTIFIER ::= { org 6 }

internet      OBJECT IDENTIFIER ::= { dod 1 }

"iso" is the root most, so make an exception for it.

Mib.xtext

OidValue:

// '::=' '{' parent=[Object] oidnum=INT '}';

'::=' '{' parent=[Object]? oidnum=INT '}';

After this, error there will be :

This file also contains a new type called MACRO.

RFC1155-SMI.mib

OBJECT-TYPE MACRO ::= BEGIN
  TYPE NOTATION ::= "SYNTAX" type (TYPE ObjectSyntax)
    "ACCESS" Access
    "STATUS" Status
  VALUE NOTATION ::= value (VALUE ObjectName)
    Access ::= "read-only"
    | "read-write"
    | "write-only"
    | "not-accessible"
    Status ::= "mandatory"
    | "optional"
    | "obsolete"
END

In order to support Macro, modify as follow:

Mib.xtext

Macro:
 name=ID 'MACRO' '::=' 'BEGIN'
 (ID 'NOTATION' '::=' MacroType+)+
 (ID '::=' (MacroList | MacroType | MacroCompound | MacroEnum) ('|' MacroType)?)*
 'END';

MacroList:
 ID '|' ID '","' ID;

MacroCompound:
 STRING? '"{"' MacroType '"}"';

MacroType:
 STRING? ID ('(' ID? mibType ')')?;

MacroEnum:
 STRING ('|' STRING)*;

RFC-1212.mib

Should display without any error.

Therefore the final Mib.xtext should look something like:

Mib.xtext

grammar com.ravi.mib.xtext.Mib hidden(WS, ML_COMMENT, SL_COMMENT)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore
generate mib "http://www.ravi.com/mib/xtext/Mib"

MibModel:
 definitions+=Definition*;

Definition:
 name=ID 'DEFINITIONS' '::=' 'BEGIN'
 Export?
 imports=Import?
 (identifiers+=Identifier | DataType | macros+=Macro)+
 'END';

Import:
 'IMPORTS' defs+=ImpDef+ ';';

ImpDef:
 objects+=MibObject (',' objects+=MibObject)* 'FROM' defname=ID;

MibObject:
 name=ID;

Export:
 'EXPORTS' ID (',' ID)* ';';

Identifier:
 ObjectType | (name=MibObject mibType value=OidValue);

ObjectType:
 name=MibObject imp=[MibObject] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
 ('DESCRIPTION' STRING)?
 ('REFERENCE' STRING)?
 ('INDEX' '{' mibType (',' mibType)* '}')?
 ('DEFVAL' '{' INT | STRING '}')?
 value=OidValue;

OidValue:
 '::=' '{' parent=[MibObject]? oidnum=INT '}';

DataType:
 ID '::=' ('[' 'APPLICATION' INT ']')? 'IMPLICIT'? (Choice | Sequence | mibType);

Choice:
 'CHOICE' '{' ID mibType (',' ID mibType)* '}';

Sequence:
 'SEQUENCE' '{' ID mibType (',' ID mibType)* '}';

mibType:
 ('SEQUENCE' 'OF')?
 ('OCTET' 'STRING' | ID | 'INTEGER' | 'OBJECT' 'IDENTIFIER')
 ('(' 'SIZE' '(' (INT '..')? INT ')' ')')?
 ('(' (INT '..')? (INT | 'MAX') ')')?
 ('{' ID '(' INT ')' (',' ID '(' INT ')')* '}')?;

Macro:
 name=ID 'MACRO' '::=' 'BEGIN'
 // TYPE NOTATION vs VALUE NOTATION
 (ID 'NOTATION' '::=' MacroType+)+
 (ID '::=' (MacroList | MacroType | MacroCompound | MacroEnum) ('|' MacroType)?)*
 'END';

MacroList:
 ID '|' ID '","' ID;

MacroCompound:
 STRING? '"{"' MacroType '"}"';

MacroType:
 STRING? ID ('(' ID? mibType ')')?;

MacroEnum:
 STRING ('|' STRING)*;

 
 /* 
  * -------------------------------------------------------------
    * Unfortunately need to overwrite following lexers
   * -------------------------------------------------------------
   */
terminal ID:
 '^'? ('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '-' | '0'..'9')*;

terminal INT returns ecore::EInt:
 ('0'..'9')+;

terminal STRING:
 '"' ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | '"'))* '"' |
 "'" ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | "'"))* "'";

terminal ML_COMMENT:
 '/*'->'*/';

terminal SL_COMMENT:
 '--' !('\n' | '\r')* ('\r'? '\n')?;

terminal WS:
 (' ' | '\t' | '\r' | '\n')+;

terminal ANY_OTHER:
 .;

TODO: 

1) Support OidValue with multiple definition "{ iso org(3) dod(6) 1 }"

Import objects should link to proper source MIB definition - Part 1

MIB file "import" really means the objects are from other MIB file (MIB Definition).

RFC1213-MIB.mib

          IMPORTS
                  mgmt, NetworkAddress, IpAddress, Counter, Gauge,
                          TimeTicks
                      FROM RFC1155-SMI
                  OBJECT-TYPE
                          FROM RFC-1212;

For example, mgmt is defined in RFC1155-SMI.mib file or OBJECT-TYPE is defined in RFC-1212.mib file.

RFC1155-SMI.mib

 mgmt          OBJECT IDENTIFIER ::= { internet 2 }

RFC-1212.mib

          OBJECT-TYPE MACRO ::=
          BEGIN
              TYPE NOTATION ::=
                                          -- must conform to
                                          -- RFC1155's ObjectSyntax
                                "SYNTAX" type(ObjectSyntax)
                                "ACCESS" Access
                                "STATUS" Status
                                DescrPart
                                ReferPart
                                IndexPart
                                DefValPart
              VALUE NOTATION ::= value (VALUE ObjectName) <etc etc>

Therefore, we need to make the references link across the file.

Xtext provides this via special attributes "importedNamespace" or "importURI". Since each MIB is enclosed by Definition block and provides Import section - its more closely resemble namespace so "importedNamespace" will be my choice.

Before working directly on MIB grammar, I decided to get familiarize on Xtext import namespace concept using 15 Minutes Tutorial example.

Notice that root rule is using multiplicity - making it single instance causes "Open-Declaration" (i.e F3) feature to stop working - I take this as hint that each files data get appended into a single model container .

Domainmodel.xtext

Domainmodel:
//  (elements += AbstractElement)*
  elements = AbstractElement
;

Therefore, the Mib grammar should be modified as follow.

Mib.xtext

MibModel:
 definitions+=Definition* ;

Definition:
 name=ID 'DEFINITIONS' '::=' 'BEGIN'
 Export?
 imports=Import?
 (identifiers+=Identifier | DataType)+
 'END';

Additionally using that 15 Minutes Tutorial again, lets modify the import syntax something similar to Mib.

blog.dmodel

//  import my.company.common.HasAuthor
  import HasAuthor from my.company.common

Therefore the grammar need to be changed as follow:

Domainmodel.xtext

Import:
//  'import' importedNamespace = QualifiedNameWithWildcard
  'import' lastSegment=(ID|'*') 'from' qname=QualifiedName
;
  
//QualifiedNameWithWildcard:
//  QualifiedName '.*'?
//;

Since "importedNamespace" attribute can't be used directly, we lose the default Xtext behaviour and need manual construct of Imported Namespace. This can be done by overriding getImportedNamespace() method from ImportedNamespaceAwareLocalScopeProvider.

DomainmodelImportedNamespaceAwareLocalScopeProvider.xtend

package org.example.domainmodel.scoping

import org.eclipse.xtext.scoping.impl.ImportedNamespaceAwareLocalScopeProvider
import org.eclipse.emf.ecore.EObject
import org.example.domainmodel.domainmodel.Import

class DomainmodelImportedNamespaceAwareLocalScopeProvider extends ImportedNamespaceAwareLocalScopeProvider {

 override String getImportedNamespace(EObject object) {
  if (object instanceof Import)
   (object as Import).qname + "." + (object as Import).lastSegment
 }

}

And to bind this class, override configureIScopeProviderDelegate() as follow:

DomainmodelRuntimeModule.java

 @Override
 public void configureIScopeProviderDelegate(com.google.inject.Binder binder) {
  binder.bind(org.eclipse.xtext.scoping.IScopeProvider.class)
    .annotatedWith(
      com.google.inject.name.Names
        .named(org.eclipse.xtext.scoping.impl.AbstractDeclarativeScopeProvider.NAMED_DELEGATE))
    .to(DomainmodelImportedNamespaceAwareLocalScopeProvider.class);
 }

Now the new import syntax should work.

TODO: Why there is another alternative ImportedNamespaceAwareLocalScopeProvider binding through Model workflow file while we can override this through above method? When or any reason at all to use this?

GenerateDomainmodel.mwe2

   fragment = scoping.ImportNamespacesScopingFragment auto-inject {}

Object Type should extends from Object Identifier

In MIB, there are 2 ways to define Identifier Objects; as Object-Identifier and Object-Type.

RFC1213-MIB.mib

          -- Object Identifier
          mib-2      OBJECT IDENTIFIER ::= { mgmt 1 }

          -- Object Type
          sysDescr OBJECT-TYPE
              SYNTAX  DisplayString (SIZE (0..255))
              ACCESS  read-only
              STATUS  mandatory
               DESCRIPTION
                      "A textual description of the entity.  This value
                      should include the full name and version
                      identification of the system's hardware type,
                      software operating-system, and networking
                      software.  It is mandatory that this only contain
                      printable ASCII characters."
              ::= { system 1 }

Object-Identifier basically contains OID name and OID value where else Object-Type contains these as well few more additional attributes.

In another word, Object-Type EClass should extend from Object-Identifier EClass. This can be done via Xtext declarative way:

Mib.xtext

Definition:
 name=ID 'DEFINITIONS' '::=' 'BEGIN'
 Export?
 imports=Import?
 (identifiers+=Identifier | DataType)+
 'END';

Object:
 name=ID;

Identifier:
 ObjectType | (name=Object mibType value=OidValue);

ObjectType:
 name=Object imp=[Object] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
 ('DESCRIPTION' STRING)?
 ('REFERENCE' STRING)?
 ('INDEX' '{' mibType (',' mibType)* '}')?
 ('DEFVAL' '{' INT | STRING '}')?
 value=OidValue;

This generates following ecore model:

Restricting auto-completion options - UI wise

Based on previous scope restriction idea, we can also try to restrict auto-completion for macro:

RFC1213-MIB.mib

          sysDescr OBJECT-TYPE
              SYNTAX  DisplayString (SIZE (0..255))
              ACCESS  read-only
              STATUS  mandatory
               DESCRIPTION
                      "A textual description of the entity.  This value
                      should include the full name and version
                      identification of the system's hardware type,
                      software operating-system, and networking
                      software.  It is mandatory that this only contain
                      printable ASCII characters."
              ::= { system 1 }

Which corresponds to following grammar:

Mib.xtext

ObjectType returns Identifier:
 name=Object imp=[Object] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
 ('DESCRIPTION' STRING)?
 ('REFERENCE' STRING)?
 ('INDEX' '{' mibType (',' mibType)* '}')?
 ('DEFVAL' '{' INT | STRING '}')?
 value=OidValue;

Currently the auto-completion showed up as :

When following scope restriction code is applied:

MibScopeProvider.xtend

 def IScope scope_Identifier_imp(Identifier id, EReference ref) {
  Scopes::scopeFor(EcoreUtil2::getAllContentsOfType(id.eContainer as Definition,
   com.ravi.mib.xtext.mib.Object).filter[it.name == "OBJECT-TYPE"])
 }

The auto-complete still lists all Objects. However, upon choosing other than "OBJECT-TYPE", Xtext highlights it as an error.

Therefore, we are half way done, now need to make sure the auto-completion list is get filtered as well, apparently that is done separately in MibProposalProvider.xtend (This is under xtext generated ui project).

MibProposalProvider.xtend

class MibProposalProvider extends AbstractMibProposalProvider { 
 override completeObjectType_Imp(EObject model, Assignment assignment, ContentAssistContext context, ICompletionProposalAcceptor acceptor) {
  lookupCrossReference(assignment.terminal as CrossReference, context, acceptor, [(EObjectOrProxy as com.ravi.mib.xtext.mib.Object).name == "OBJECT-TYPE"])
 }
}

Now all works as it should be.

Few clarifications:

• Should not make "OBJECT-TYPE" directly as keyword in Xtext as this is used as import. Sure, we can look for a way to overcome this i.e accepting as identifier or keyword based on context, however "OBJECT-TYPE" is rightfully just a macro and not a keyword in ASN.1 (Mib syntax is based on this). Therefore, I prefer to leave it as identifier.
• I am not sure why auto-completion options for OidValue's parent works without the need to create similiar completeOidValue_parent() method. Hope someone can enlighten me on this.
• Notice that in this round in method "scope_Identifier_imp", I have used "Scopes::scopeFor(...)" instead of  "new SimpleScope(Scopes::scopedElementsFor(...)". The latter gives us access to EObjectDescription (see also scopedElementsFor() with custom function argument to compute element name). However, for our case - no name customization is done, former is ideal.

Restricting auto-completion options - Scope wise

In MIB file, 2 values are required for oid value; parent object and object id. As for parent object name, editor should able to provide list of auto-completion based on previously defined object identifier plus objects that are imported from other MIB file.

The problem with this imported objects are not all are of object identifiers, some can be macro (OBJECT-TYPE) or some can be data types (NetworkAddress, IpAddress, Counter, Gauge etc). Hence its wrong to list all these as possible parent objects within the auto-completion list.

The screenshot illustrates the scenario. Therefore the challenge is to remove those options (macro and datatype) from the option list.

In order to do this, Xtext provides scope restriction. Since I have not (yet) look into cross reference with other MIB files, I will just be happy by hard-coding the exclusion list. This is temporary until I dive into MIB exports from dependency files.

While searching over net for the answer, Generally, I realized few things:

• Unfortunately this is no longer declarative codes but need to be told imperatively - requires some 'coding'. We will use Xtend.
• Xtend is something rather new in Xtext, therefore quite number of examples out there may use Java.
• While Xtend simplify the expression, you may need to spend some time reading on Xtend for more effective coding!

Here is the code I wrote this achieve this:

MibScopeProvider.xtend

package com.ravi.mib.xtext.scoping

import org.eclipse.xtext.scoping.IScope
import com.ravi.mib.xtext.mib.OidValue
import org.eclipse.emf.ecore.EReference
import org.eclipse.xtext.scoping.Scopes
import com.ravi.mib.xtext.mib.Definition
import org.eclipse.xtext.EcoreUtil2
import org.eclipse.xtext.scoping.impl.SimpleScope
import java.util.HashSet

/**
 * This class contains custom scoping description.
 * 
 * see : http://www.eclipse.org/Xtext/documentation.html#scoping
 * on how and when to use it 
 *
 */
class MibScopeProvider extends org.eclipse.xtext.scoping.impl.AbstractDeclarativeScopeProvider {
 def IScope scope_OidValue_parent(OidValue value, EReference ref) {
  val excluded = new HashSet(#["NetworkAddress", "IpAddress", "Counter", "Gauge", "TimeTicks", "OBJECT-TYPE"])
  new SimpleScope(
   Scopes.scopedElementsFor(
    EcoreUtil2::getAllContentsOfType(value.eContainer.eContainer as Definition,
     com.ravi.mib.xtext.mib.Object).filter[!excluded.contains(it.name)]));
 }
}

MibScopeProvider.xtend is a xtend file pre-generated when Xtext project was created. I find it rather interesting that method name has lot to do with context where it will be called. "scope_OidValue_parent", "OidValue" is the related EObject for the rule and "parent" is EReference that we are interested.

Here is the complement rule to go with the code:

Mib.xtext

OidValue:
 '::=' '{' parent=[Object] oidnum=INT '}';

Here is the screenshot once this implemented:

Now for fun lets compare Xtend code vs Java code that generated from it, just to show arguably how much saving + readability Xtend can provide over Java.

Bash Terminal

dev@ravi-vm:~/workspace/com.ravi.mib.xtext$ wc ./src/com/ravi/mib/xtext/scoping/MibScopeProvider.xtend
  30   78 1030 ./src/com/ravi/mib/xtext/scoping/MibScopeProvider.xtend
dev@ravi-vm:~/workspace/com.ravi.mib.xtext$ wc ./xtend-gen/com/ravi/mib/xtext/scoping/MibScopeProvider.java
  52  153 2362 ./xtend-gen/com/ravi/mib/xtext/scoping/MibScopeProvider.java

Its almost half less coding!

Comma separated list

There are few places in MIB sytax that allows comma separated list (we can even generalized it by some character delimited list), here is one snippet of rtc1213 mib file.

RFC1213-MIB.mib

          IMPORTS
                  mgmt, NetworkAddress, IpAddress, Counter, Gauge,
                          TimeTicks
                      FROM RFC1155-SMI
                  OBJECT-TYPE
                          FROM RFC-1212;

For this I have used following parsing rules:

Mib.xtext

Import:
 'IMPORTS' defs+=ImpDef+ ';';

ImpDef:
 (objects+=Object ','?)+ 'FROM' def=ID;

Object:
 name=ID;

I have noticed quite a few tutorial sites would write it as

Mib.xtext

Import:
 'IMPORTS' defs+=ImpDef+ ';';

ImpDef:
 objects+=Object (',' objects+=Object)* 'FROM' def=ID;

Object:
 name=ID;

And they are right, my rule even thought looks simple but its not bullet proof; in fact it allows incorrect syntax - "mgmt NetworkAddress ...", I have fixed these occurrences through out my grammar file (6 places), next time when I add grammar snippet, you will noticed it of the updated version.

Parse MIB file

This is the initial Xtext grammar that I wrote to load RFC1213-MIB. You can get a copy of RFC1213-MIB from rfc1213.txt, extract content from Section 6: "Definitions" and save it as RFC1213-MIB.mib.

The grammar is product of few iterations of improvement, hence some rules may not be straight forward, I will try to explain the reason as best I can.

Project Info

An Xtext project was created with following attributes:

• Project Name: com.ravi.mib.xtext
• Language Name : com.ravi.mib.xtext.Mib
• Language extensions: mib

Eclipse Modeling Tools Version: Luna Release (4.4.0)

Xtext SDK Version: 2.6.0

Overwrite the generated Mib.xtext with following content:

Mib.xtext

grammar com.ravi.mib.xtext.Mib hidden(WS, ML_COMMENT, SL_COMMENT)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore
generate mib "http://www.ravi.com/mib/xtext/Mib"

Definition:
 name=ID 'DEFINITIONS' '::=' 'BEGIN'
 Export?
 imports=Import?
 (identifiers+=ObjectType | identifiers+=Identifier | DataType)+
 'END';

Import:
 'IMPORTS' defs+=ImpDef+ ';';

ImpDef:
 (objects+=Object ','?)+ 'FROM' def=ID;

Object:
 name=ID;

Export:
 'EXPORTS' (ID ','?)+ ';';

ObjectType returns Identifier:
 name=Object imp=[Object] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
 ('DESCRIPTION' STRING)?
 ('REFERENCE' STRING)?
 ('INDEX' '{' (mibType ','?)+ '}')?
 ('DEFVAL' '{' INT | STRING '}')?
 value=OidValue;

Identifier returns Identifier:
 name=Object mibType value=OidValue;

OidValue:
 '::=' '{' parent=[Object] oidnum=INT '}';

DataType:
 ID '::=' ('[' 'APPLICATION' INT ']')? 'IMPLICIT'? (Choice | Sequence | mibType);

Choice:
 'CHOICE' '{' (ID mibType ','?)+ '}';

Sequence:
 'SEQUENCE' '{' (ID mibType ','?)+ '}';

mibType:
 ('SEQUENCE' 'OF')?
 ('OCTET' 'STRING' | ID | 'INTEGER')
 ('(' 'SIZE'? '('? (INT '..')? INT ')'? ')')?
 ('{' (ID '(' INT ')' ','?)+ '}')? |
 'OBJECT' 'IDENTIFIER';

 /* 
  * -------------------------------------------------------------
    * Unfortunately need to overwrite following lexers
   * -------------------------------------------------------------
   */
terminal ID:
 '^'? ('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '-' | '0'..'9')*;

terminal INT returns ecore::EInt:
 ('0'..'9')+;

terminal STRING:
 '"' ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | '"'))* '"' |
 "'" ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | "'"))* "'";

terminal ML_COMMENT:
 '/*'->'*/';

terminal SL_COMMENT:
 '--' !('\n' | '\r')* ('\r'? '\n')?;

terminal WS:
 (' ' | '\t' | '\r' | '\n')+;

terminal ANY_OTHER:
 .;

Snapshot of the editor

The above grammar produces pretty neat editor, with nice syntax highlighting, outline and somewhat limited auto-completion.

Some explanations on the grammar:

Ecore

grammar com.ravi.mib.xtext.Mib hidden(WS, ML_COMMENT, SL_COMMENT)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore
generate mib "http://www.ravi.com/mib/xtext/Mib"

This line tells Xtext to generate Ecore model on its own

Terminal Rules

grammar com.ravi.mib.xtext.Mib hidden(WS, ML_COMMENT, SL_COMMENT)

 /* 
  * -------------------------------------------------------------
    * Unfortunately need to overwrite following lexers
   * -------------------------------------------------------------
   */
terminal ID:
 '^'? ('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '-' | '0'..'9')*;

terminal INT returns ecore::EInt:
 ('0'..'9')+;

terminal STRING:
 '"' ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | '"'))* '"' |
 "'" ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | "'"))* "'";

terminal ML_COMMENT:
 '/*'->'*/';

terminal SL_COMMENT:
 '--' !('\n' | '\r')* ('\r'? '\n')?;

terminal WS:
 (' ' | '\t' | '\r' | '\n')+;

terminal ANY_OTHER:
 .;

Xtext generates grammar by extending "org.eclipse.xtext.common.Terminals" which defines most common terminal rules (lexer), however in this case, I had to take over in order to overwrite

• ID - In MIB, its common for identifier to use hyphen character.
• SL_COMMENT - MIB uses '--' as comment.

Note that we need to specify hidden() as well, this tells Xtext to ignore those terminal rules on parsing, this keeps parsing stage clean from these noise i.e can build parser rules as these hidden terminal rules never existed.

Overview

Introduction

I have recently started looking at various DSL technologies out there to find best fit for my use. With some knowledge on EMF couples years back, Xtext has impressed me the most. I still have some learning curve to go through before I can convince my self this is the tool for me.

The way I want to do this is by picking up one rather medium to complex DSL out there (Its always good to pick up a mature DSL with years of real use in the field) and prove Xtext about to handle it in flying colors!

My choice of DSL for this use-case is MIB. Basic objective is to open any given mib file with full syntax highlighting, content assist (pretending we can edit it) plus it should resolve imports from its dependency MIB  files (definitions). And to make it more interesting, finally it should generate list of oid object by name, full dotted notation, derived data type and its primitive data type.

Target

The real target of this is rather for me to go journal my learning process on Xtext - not so much on MIB Parser - but it will be a bonus if get it done as well. 

There are some very good documentations, tutorials on Xtext out there and I don't intent this to be called any of those. I rather would like to invite everyone to share and feedback on each of the entries that I have added to make this learning more fun plus hopefully will benefit others along the way.

That said, I would like to add a disclaimer that this code may not be production worthy and take it with a pinch of salt.