Line of Codes

Now the essential part of MIB Parser is completed and achieved its objective.

Here is LoC required to get it to current state.

• Xtext (134 lines)
• Xcore ( 107 lines)
• Java (6 lines)
• Xtend (27 lines)
• Acceleo (31 lines)

Some of the xtend codes that I written earlier for scoping is not counted as its deemed as not required once I understand namespace concept in Xtext.

Completing Data Type reference in Xtext Grammar

As described in overview from the first post, its almost close to produce CSV artefacts containing OID short name, Oid in dot notation number, derived data type and its primitive data type.

The part missing from model now is the data type information. This can be completed with following modification to Xtext grammar and Xcore.

Mib.xtext

ObjectType:
 name=ID imp=[Macro] 
 'SYNTAX' type=mibType2
 'ACCESS' access=AccessEnum 
 'STATUS' status=StatusEnum 
 ('DESCRIPTION' description=STRING)? 
 ('REFERENCE' reference=STRING)? 
 ('INDEX' '{' mibType (',' mibType)* '}')? 
 ('DEFVAL' '{' INT | STRING '}')? 
 value=OidValue;

ObjectType class will now contain feature type of mibType2 among others. mibType2 is similiar to mibType, the reason I have created another instance of mibType is as it used in lot of places such as Macro, Object Type Index, Choice etc. All these rightfully need to be changed as well, however for now I want to keep it simple - it can be overwhelming otherwise.

Mib.xtext

mibType2:
 (sequence?=SequenceOf)? 
 dataType=DataType
 ('(' 'SIZE' '(' (INT '..')? INT ')'')')? 
 ('(' (INT '..')? (INT | 'MAX') ')')? 
 ('{' ID '(' INT ')' (',' ID '(' INT ')')* '}')?;

DataType:
 name=(OctectString | 'INTEGER' | ObjectIdentifier) |
 derived=[TypeDefinition];

SequenceOf:
 'SEQUENCE' 'OF';
 
ObjectIdentifier:
 'OBJECT' 'IDENTIFIER';

OctectString:
 'OCTET' 'STRING';

One trick I learned here is that if there are multiple keywords make up a syntax, its better to push those down to its own rule e.g SequenceOf - Xtext  intelligent enough to know this is terminal like rule and will not create an EClass for it.

Datatype takes primitive data types (3 types) or a derived data type from TypeDefinition.  Note that this 'TypeDefinition' used to be called 'DataType' earlier - re-factored to reflect semantic better.

Mib.xtext

Definition:
 name=ID 'DEFINITIONS' '::=' 'BEGIN'
 Export?
 imports=Import?
 (identifiers+=Identifier | typedef+=TypeDefinition | macros+=Macro)+
 'END';

TypeDefinition:
 name=ID '::=' ('[' 'APPLICATION' INT ']')? 
 'IMPLICIT'? (Choice | Sequence | type=mibType2);

And to reflect this on Xcore, following need to be changed:

Mib.xtext

class Definition {
 String name
 contains Import imports
 contains Identifier[] identifiers
 contains TypeDefinition[] typedef
 contains Macro[] macros
}

class TypeDefinition {
 String name
 contains mibType2 ^type
}

Setting TypeDefinition as containment for Definition allows Object Type's type to refer to this.

Mib.xtext

class ObjectType extends Identifier {
 refers Macro imp
 contains mibType2 ^type
 AccessEnum access
 StatusEnum status
 String description
 String reference
}

enum AccessEnum {
 readOnly as "read-only"
 readWrite as "read-write" = 1
 writeOnly as "write-only" = 2
 notAccessible as "not-accessible" = 3
}

enum StatusEnum {
 mandatory
 optional = 1
 obsolete = 2
 deprecated = 3
}

Now running this will give full "open declaration" for type with derived data type as well.

Import objects should link to proper source MIB definition - Part 3

From last Mib.xtext, the main change need to be done is to break Mib Objects from Import Objects. i.e The Mib Object identifier does not need to directly point to import section's identifier. Import works more as 'namespace' lookup for each identifier. Therefore Mib grammar should be modified as follow:

Mib.xtext

Import:
 name='IMPORTS' defs+=ImpDef+ ';';

ImpDef:
 objects+=ImpObject (',' objects+=ImpObject)* 'FROM' name=ID;

ImpObject:
 name=ID;

And re-point imp to macro as well as simplify Identifier:

Mib.xtext

Identifier:
 ObjectType | (name=ID mibType value=OidValue);
// ObjectType | (name=MibObject mibType value=OidValue);

ObjectType:
 name=ID imp=[Macro] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
// name=MibObject imp=[MibObject] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
 ('DESCRIPTION' STRING)?
 ('REFERENCE' STRING)?
 ('INDEX' '{' mibType (',' mibType)* '}')?
 ('DEFVAL' '{' INT | STRING '}')?
 value=OidValue;

OidValue:
 '::=' '{' parent=[Identifier]? oidnum=INT '}';
// '::=' '{' parent=[MibObject]? oidnum=INT '}';

And since we are not using "importedNamespace" attribute, we need to complete the namespace by overriding "internalGetImportedNamespaceResolvers()" - note that I used "getImportedNamespace()" on earlier example - but that can't be used here as Xtext not able to know all the imported object since the structure is pretty complex in MIB. Hence overriding the "internalGetImportedNamespaceResolvers()" which is in fact the caller for "getImportedNamespace()" is more appropriate here.

MibImportedNamespaceAwareLocalScopeProvider.xtend

class MibImportedNamespaceAwareLocalScopeProvider extends ImportedNamespaceAwareLocalScopeProvider {
 override List internalGetImportedNamespaceResolvers(EObject context, boolean ignoreCase) {
  val list = new LinkedList
  if (context instanceof Definition) {
   if ((context as Definition).imports != null) {
    for (x : EcoreUtil2.getAllContentsOfType((context as Definition).imports, ImpObject)) {
     val qname = (x.eContainer as ImpDef).name + "." + x.name
     list.add(createImportedNamespaceResolver(qname, ignoreCase));
    }
   }
  } 
  list
 }
}

And of course bind this class by:

MibRuntimeModule.java

public class MibRuntimeModule extends
  com.ravi.mib.xtext.AbstractMibRuntimeModule {
 @Override
 public void configureIScopeProviderDelegate(com.google.inject.Binder binder) {
  binder.bind(org.eclipse.xtext.scoping.IScopeProvider.class)
    .annotatedWith(
      com.google.inject.name.Names
        .named(org.eclipse.xtext.scoping.impl.AbstractDeclarativeScopeProvider.NAMED_DELEGATE))
    .to(MibImportedNamespaceAwareLocalScopeProvider.class);
 }
}

Now, "Open-Declaration" (i.e F3) feature should work for both "Mib object identifier" and "OBJECT-TYPE" macro, open up file that contains those definitions.

Import objects should link to proper source MIB definition - Part 2

Now will apply the same changes to Mib project.

As preparation, need to get all dependent MIB files:

• RFC1155-SMI.mib (Extract from rfc, section 6 - Definitions)
• RFC-1212.mib (Extract from rfc, section 4 - Defining Objects, need to complete it manually)
• RFC1158-MIB.mib (Extract from rfc, section 6 - Definitions) - This is dependent for RFC-1212 on "DisplayString" definition only. This is obsoleted by RFC1213.

Open each files in Xtext Editor, make sure MIB grammar able to handle them.

 

RFC1155-SMI.mib

Following format is not supported:

RFC1155-SMI.mib

internet      OBJECT IDENTIFIER ::= { iso org(3) dod(6) 1 }

Therefore, modify the file to following

RFC1155-SMI.mib

-- internet      OBJECT IDENTIFIER ::= { iso org(3) dod(6) 1 }

iso           OBJECT IDENTIFIER ::= { 0 }

org           OBJECT IDENTIFIER ::= { iso 3 }

dod           OBJECT IDENTIFIER ::= { org 6 }

internet      OBJECT IDENTIFIER ::= { dod 1 }

"iso" is the root most, so make an exception for it.

Mib.xtext

OidValue:

// '::=' '{' parent=[Object] oidnum=INT '}';

'::=' '{' parent=[Object]? oidnum=INT '}';

After this, error there will be :

This file also contains a new type called MACRO.

RFC1155-SMI.mib

OBJECT-TYPE MACRO ::= BEGIN
  TYPE NOTATION ::= "SYNTAX" type (TYPE ObjectSyntax)
    "ACCESS" Access
    "STATUS" Status
  VALUE NOTATION ::= value (VALUE ObjectName)
    Access ::= "read-only"
    | "read-write"
    | "write-only"
    | "not-accessible"
    Status ::= "mandatory"
    | "optional"
    | "obsolete"
END

In order to support Macro, modify as follow:

Mib.xtext

Macro:
 name=ID 'MACRO' '::=' 'BEGIN'
 (ID 'NOTATION' '::=' MacroType+)+
 (ID '::=' (MacroList | MacroType | MacroCompound | MacroEnum) ('|' MacroType)?)*
 'END';

MacroList:
 ID '|' ID '","' ID;

MacroCompound:
 STRING? '"{"' MacroType '"}"';

MacroType:
 STRING? ID ('(' ID? mibType ')')?;

MacroEnum:
 STRING ('|' STRING)*;

RFC-1212.mib

Should display without any error.

Therefore the final Mib.xtext should look something like:

Mib.xtext

grammar com.ravi.mib.xtext.Mib hidden(WS, ML_COMMENT, SL_COMMENT)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore
generate mib "http://www.ravi.com/mib/xtext/Mib"

MibModel:
 definitions+=Definition*;

Definition:
 name=ID 'DEFINITIONS' '::=' 'BEGIN'
 Export?
 imports=Import?
 (identifiers+=Identifier | DataType | macros+=Macro)+
 'END';

Import:
 'IMPORTS' defs+=ImpDef+ ';';

ImpDef:
 objects+=MibObject (',' objects+=MibObject)* 'FROM' defname=ID;

MibObject:
 name=ID;

Export:
 'EXPORTS' ID (',' ID)* ';';

Identifier:
 ObjectType | (name=MibObject mibType value=OidValue);

ObjectType:
 name=MibObject imp=[MibObject] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
 ('DESCRIPTION' STRING)?
 ('REFERENCE' STRING)?
 ('INDEX' '{' mibType (',' mibType)* '}')?
 ('DEFVAL' '{' INT | STRING '}')?
 value=OidValue;

OidValue:
 '::=' '{' parent=[MibObject]? oidnum=INT '}';

DataType:
 ID '::=' ('[' 'APPLICATION' INT ']')? 'IMPLICIT'? (Choice | Sequence | mibType);

Choice:
 'CHOICE' '{' ID mibType (',' ID mibType)* '}';

Sequence:
 'SEQUENCE' '{' ID mibType (',' ID mibType)* '}';

mibType:
 ('SEQUENCE' 'OF')?
 ('OCTET' 'STRING' | ID | 'INTEGER' | 'OBJECT' 'IDENTIFIER')
 ('(' 'SIZE' '(' (INT '..')? INT ')' ')')?
 ('(' (INT '..')? (INT | 'MAX') ')')?
 ('{' ID '(' INT ')' (',' ID '(' INT ')')* '}')?;

Macro:
 name=ID 'MACRO' '::=' 'BEGIN'
 // TYPE NOTATION vs VALUE NOTATION
 (ID 'NOTATION' '::=' MacroType+)+
 (ID '::=' (MacroList | MacroType | MacroCompound | MacroEnum) ('|' MacroType)?)*
 'END';

MacroList:
 ID '|' ID '","' ID;

MacroCompound:
 STRING? '"{"' MacroType '"}"';

MacroType:
 STRING? ID ('(' ID? mibType ')')?;

MacroEnum:
 STRING ('|' STRING)*;

 
 /* 
  * -------------------------------------------------------------
    * Unfortunately need to overwrite following lexers
   * -------------------------------------------------------------
   */
terminal ID:
 '^'? ('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '-' | '0'..'9')*;

terminal INT returns ecore::EInt:
 ('0'..'9')+;

terminal STRING:
 '"' ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | '"'))* '"' |
 "'" ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | "'"))* "'";

terminal ML_COMMENT:
 '/*'->'*/';

terminal SL_COMMENT:
 '--' !('\n' | '\r')* ('\r'? '\n')?;

terminal WS:
 (' ' | '\t' | '\r' | '\n')+;

terminal ANY_OTHER:
 .;

TODO: 

1) Support OidValue with multiple definition "{ iso org(3) dod(6) 1 }"

Import objects should link to proper source MIB definition - Part 1

MIB file "import" really means the objects are from other MIB file (MIB Definition).

RFC1213-MIB.mib

          IMPORTS
                  mgmt, NetworkAddress, IpAddress, Counter, Gauge,
                          TimeTicks
                      FROM RFC1155-SMI
                  OBJECT-TYPE
                          FROM RFC-1212;

For example, mgmt is defined in RFC1155-SMI.mib file or OBJECT-TYPE is defined in RFC-1212.mib file.

RFC1155-SMI.mib

 mgmt          OBJECT IDENTIFIER ::= { internet 2 }

RFC-1212.mib

          OBJECT-TYPE MACRO ::=
          BEGIN
              TYPE NOTATION ::=
                                          -- must conform to
                                          -- RFC1155's ObjectSyntax
                                "SYNTAX" type(ObjectSyntax)
                                "ACCESS" Access
                                "STATUS" Status
                                DescrPart
                                ReferPart
                                IndexPart
                                DefValPart
              VALUE NOTATION ::= value (VALUE ObjectName) <etc etc>

Therefore, we need to make the references link across the file.

Xtext provides this via special attributes "importedNamespace" or "importURI". Since each MIB is enclosed by Definition block and provides Import section - its more closely resemble namespace so "importedNamespace" will be my choice.

Before working directly on MIB grammar, I decided to get familiarize on Xtext import namespace concept using 15 Minutes Tutorial example.

Notice that root rule is using multiplicity - making it single instance causes "Open-Declaration" (i.e F3) feature to stop working - I take this as hint that each files data get appended into a single model container .

Domainmodel.xtext

Domainmodel:
//  (elements += AbstractElement)*
  elements = AbstractElement
;

Therefore, the Mib grammar should be modified as follow.

Mib.xtext

MibModel:
 definitions+=Definition* ;

Definition:
 name=ID 'DEFINITIONS' '::=' 'BEGIN'
 Export?
 imports=Import?
 (identifiers+=Identifier | DataType)+
 'END';

Additionally using that 15 Minutes Tutorial again, lets modify the import syntax something similar to Mib.

blog.dmodel

//  import my.company.common.HasAuthor
  import HasAuthor from my.company.common

Therefore the grammar need to be changed as follow:

Domainmodel.xtext

Import:
//  'import' importedNamespace = QualifiedNameWithWildcard
  'import' lastSegment=(ID|'*') 'from' qname=QualifiedName
;
  
//QualifiedNameWithWildcard:
//  QualifiedName '.*'?
//;

Since "importedNamespace" attribute can't be used directly, we lose the default Xtext behaviour and need manual construct of Imported Namespace. This can be done by overriding getImportedNamespace() method from ImportedNamespaceAwareLocalScopeProvider.

DomainmodelImportedNamespaceAwareLocalScopeProvider.xtend

package org.example.domainmodel.scoping

import org.eclipse.xtext.scoping.impl.ImportedNamespaceAwareLocalScopeProvider
import org.eclipse.emf.ecore.EObject
import org.example.domainmodel.domainmodel.Import

class DomainmodelImportedNamespaceAwareLocalScopeProvider extends ImportedNamespaceAwareLocalScopeProvider {

 override String getImportedNamespace(EObject object) {
  if (object instanceof Import)
   (object as Import).qname + "." + (object as Import).lastSegment
 }

}

And to bind this class, override configureIScopeProviderDelegate() as follow:

DomainmodelRuntimeModule.java

 @Override
 public void configureIScopeProviderDelegate(com.google.inject.Binder binder) {
  binder.bind(org.eclipse.xtext.scoping.IScopeProvider.class)
    .annotatedWith(
      com.google.inject.name.Names
        .named(org.eclipse.xtext.scoping.impl.AbstractDeclarativeScopeProvider.NAMED_DELEGATE))
    .to(DomainmodelImportedNamespaceAwareLocalScopeProvider.class);
 }

Now the new import syntax should work.

TODO: Why there is another alternative ImportedNamespaceAwareLocalScopeProvider binding through Model workflow file while we can override this through above method? When or any reason at all to use this?

GenerateDomainmodel.mwe2

   fragment = scoping.ImportNamespacesScopingFragment auto-inject {}