Showing posts with label Scope Restriction. Show all posts
Showing posts with label Scope Restriction. Show all posts

Friday, 3 October 2014

Generate Oid list from Mib model

To generate Oid list (i.e text file) from Mib model, there are a few readily available Model to Text Transformation (M2T) tools for EMF.
I have started this work on Acceleo, and find the editor's content assist is pretty good and and straight forward way to apply launcher menu for mib files - almost no coding except the template declaration which is based on MOFM2T.

I will try to detail the steps clearly as I could for others to benefit. I struggled to get these end to end steps correct as most of tutorials out there are by features and less based on use-case (on top of that I used Xcore for model, hence it was a blocker at start).
  1. Install Acceleo from Modelling Component place (Version: 3.5.1), and change to Acceleo perspective, this makes lots of handy tool/wizard simple to find as oppose continue working on Java perspective.
  2. Acceleo Create Project wizard does not recognize Xcore as model. Therefore, I had to generate Ecore model first. Choose Mib.xcore from Package Explorer, and click "new -> Other..", choose EMF Generator Model. This will generate both Ecore and genmodel.
  3. Create a new Acceleo project "com.ravi.mib.acceleo.module.oid". On second wizard page - "Create a new Acceleo generation file", change module name to "oid" and search for Mib model from "Runtime Version" (http://www.ravi.com/ibm/xtext/Mib). Change template name to "generateOid" and type to "MibModel"
  4. Once the project created, edit com/ravi/mib/acceleo/module/oid/common/oid.mtl file and apply following template.
  5. oid.mtl
    [comment encoding = UTF-8 /]
    [module oid('http://www.ravi.com/mib/xtext/Mib')]
    
    [template public generateOID(model : MibModel)]
     [comment @main /]
     [for (d : Definition | model.definitions)]
      [file (d.name.toString().concat('.oid'), false, 'UTF-8')]
       [for (oid : Identifier | d.identifiers)]
        [if (isValidObjectType(oid))]
    [oid.name/],[oid.value.oid/],[getPrimitive(oid)/],[getDerived(oid)/]
        [/if]
       [/for]
      [/file]
     [/for]
    [/template]
    
    [query private isValidObjectType(oid : Identifier) : Boolean = 
     oid.oclIsTypeOf(ObjectType) and
     not oid.oclAsType(ObjectType).type.typeName.oclIsUndefined() and
     oid.oclAsType(ObjectType).type.typeName.toString().strcmp('OBJECT IDENTIFIER') <> 0
    /]
    
    [query private getPrimitive(oid : Identifier) : String =
      if oid.oclAsType(ObjectType).type.dataType.derived.name.oclIsUndefined() then
      ''
     else
      oid.oclAsType(ObjectType).type.dataType.derived.name.toString()
     endif
    /]
    
    [query private getDerived(oid : Identifier) : String = 
     oid.oclAsType(ObjectType).type.typeName.toString()
    /]
    

  6. There will be an error afterwards for "oid.common" package to be exported. Do as suggested.
  7. Now for launcher menu, select earlier created Acceleo project from Package Explorer and choose new Acceleo UI Launcher project, On "Acceleo UI Launcher settings", change model file name filter to "*.mib" and target folder access to "...getFolder("oid")".
  8. Finally modify label for menu as "Mib Generator"
  9. Once eclipse instance is launched, Generate Oid option should available for mib files.
RFC1213-MIB.oid
sysDescr,1.3.6.1.2.1.1.1,DisplayString,OCTET STRING
sysUpTime,1.3.6.1.2.1.1.3,TimeTicks,INTEGER
sysContact,1.3.6.1.2.1.1.4,DisplayString,OCTET STRING
sysName,1.3.6.1.2.1.1.5,DisplayString,OCTET STRING
sysLocation,1.3.6.1.2.1.1.6,DisplayString,OCTET STRING
sysServices,1.3.6.1.2.1.1.7,,INTEGER
ifNumber,1.3.6.1.2.1.2.1,,INTEGER
ifIndex,1.3.6.1.2.1.2.2.1.1,,INTEGER
ifDescr,1.3.6.1.2.1.2.2.1.2,DisplayString,OCTET STRING
ifType,1.3.6.1.2.1.2.2.1.3,,INTEGER
ifMtu,1.3.6.1.2.1.2.2.1.4,,INTEGER
ifSpeed,1.3.6.1.2.1.2.2.1.5,Gauge,INTEGER
ifPhysAddress,1.3.6.1.2.1.2.2.1.6,PhysAddress,OCTET STRING
ifAdminStatus,1.3.6.1.2.1.2.2.1.7,,INTEGER
ifOperStatus,1.3.6.1.2.1.2.2.1.8,,INTEGER
ifLastChange,1.3.6.1.2.1.2.2.1.9,TimeTicks,INTEGER
ifInOctets,1.3.6.1.2.1.2.2.1.10,Counter,INTEGER
ifInUcastPkts,1.3.6.1.2.1.2.2.1.11,Counter,INTEGER
ifInNUcastPkts,1.3.6.1.2.1.2.2.1.12,Counter,INTEGER
ifInDiscards,1.3.6.1.2.1.2.2.1.13,Counter,INTEGER
ifInErrors,1.3.6.1.2.1.2.2.1.14,Counter,INTEGER
ifInUnknownProtos,1.3.6.1.2.1.2.2.1.15,Counter,INTEGER
ifOutOctets,1.3.6.1.2.1.2.2.1.16,Counter,INTEGER
ifOutUcastPkts,1.3.6.1.2.1.2.2.1.17,Counter,INTEGER
ifOutNUcastPkts,1.3.6.1.2.1.2.2.1.18,Counter,INTEGER
ifOutDiscards,1.3.6.1.2.1.2.2.1.19,Counter,INTEGER
ifOutErrors,1.3.6.1.2.1.2.2.1.20,Counter,INTEGER
ifOutQLen,1.3.6.1.2.1.2.2.1.21,Gauge,INTEGER
atIfIndex,1.3.6.1.2.1.3.1.1.1,,INTEGER
TODO: Is there any way for Xcore to register its model URI so that Acceleo able to recognize without going through Ecore generation?

TODO: How to limit OID list down to the selected mib file alone?

Tuesday, 30 September 2014

Completing Data Type reference in Xtext Grammar

As described in overview from the first post, its almost close to produce CSV artefacts containing OID short name, Oid in dot notation number, derived data type and its primitive data type.

The part missing from model now is the data type information. This can be completed with following modification to Xtext grammar and Xcore.


Mib.xtext
ObjectType:
 name=ID imp=[Macro] 
 'SYNTAX' type=mibType2
 'ACCESS' access=AccessEnum 
 'STATUS' status=StatusEnum 
 ('DESCRIPTION' description=STRING)? 
 ('REFERENCE' reference=STRING)? 
 ('INDEX' '{' mibType (',' mibType)* '}')? 
 ('DEFVAL' '{' INT | STRING '}')? 
 value=OidValue;

ObjectType class will now contain feature type of mibType2 among others. mibType2 is similiar to mibType, the reason I have created another instance of mibType is as it used in lot of places such as Macro, Object Type Index, Choice etc. All these rightfully need to be changed as well, however for now I want to keep it simple - it can be overwhelming otherwise.


Mib.xtext
mibType2:
 (sequence?=SequenceOf)? 
 dataType=DataType
 ('(' 'SIZE' '(' (INT '..')? INT ')'')')? 
 ('(' (INT '..')? (INT | 'MAX') ')')? 
 ('{' ID '(' INT ')' (',' ID '(' INT ')')* '}')?;

DataType:
 name=(OctectString | 'INTEGER' | ObjectIdentifier) |
 derived=[TypeDefinition];

SequenceOf:
 'SEQUENCE' 'OF';
 
ObjectIdentifier:
 'OBJECT' 'IDENTIFIER';

OctectString:
 'OCTET' 'STRING';


One trick I learned here is that if there are multiple keywords make up a syntax, its better to push those down to its own rule e.g SequenceOf - Xtext  intelligent enough to know this is terminal like rule and will not create an EClass for it.

Datatype takes primitive data types (3 types) or a derived data type from TypeDefinition.  Note that this 'TypeDefinition' used to be called 'DataType' earlier - re-factored to reflect semantic better.


Mib.xtext
Definition:
 name=ID 'DEFINITIONS' '::=' 'BEGIN'
 Export?
 imports=Import?
 (identifiers+=Identifier | typedef+=TypeDefinition | macros+=Macro)+
 'END';

TypeDefinition:
 name=ID '::=' ('[' 'APPLICATION' INT ']')? 
 'IMPLICIT'? (Choice | Sequence | type=mibType2);

And to reflect this on Xcore, following need to be changed:


Mib.xtext
class Definition {
 String name
 contains Import imports
 contains Identifier[] identifiers
 contains TypeDefinition[] typedef
 contains Macro[] macros
}

class TypeDefinition {
 String name
 contains mibType2 ^type
}

Setting TypeDefinition as containment for Definition allows Object Type's type to refer to this.

Mib.xtext
class ObjectType extends Identifier {
 refers Macro imp
 contains mibType2 ^type
 AccessEnum access
 StatusEnum status
 String description
 String reference
}

enum AccessEnum {
 readOnly as "read-only"
 readWrite as "read-write" = 1
 writeOnly as "write-only" = 2
 notAccessible as "not-accessible" = 3
}

enum StatusEnum {
 mandatory
 optional = 1
 obsolete = 2
 deprecated = 3
}

Now running this will give full "open declaration" for type with derived data type as well.


Tuesday, 9 September 2014

Restricting auto-completion options - UI wise

Based on previous scope restriction idea, we can also try to restrict auto-completion for macro:
RFC1213-MIB.mib
          sysDescr OBJECT-TYPE
              SYNTAX  DisplayString (SIZE (0..255))
              ACCESS  read-only
              STATUS  mandatory
               DESCRIPTION
                      "A textual description of the entity.  This value
                      should include the full name and version
                      identification of the system's hardware type,
                      software operating-system, and networking
                      software.  It is mandatory that this only contain
                      printable ASCII characters."
              ::= { system 1 }
Which corresponds to following grammar:
Mib.xtext
ObjectType returns Identifier:
 name=Object imp=[Object] 'SYNTAX' mibType 'ACCESS' ID 'STATUS' ID
 ('DESCRIPTION' STRING)?
 ('REFERENCE' STRING)?
 ('INDEX' '{' mibType (',' mibType)* '}')?
 ('DEFVAL' '{' INT | STRING '}')?
 value=OidValue;
Currently the auto-completion showed up as :


When following scope restriction code is applied:


MibScopeProvider.xtend
 def IScope scope_Identifier_imp(Identifier id, EReference ref) {
  Scopes::scopeFor(EcoreUtil2::getAllContentsOfType(id.eContainer as Definition,
   com.ravi.mib.xtext.mib.Object).filter[it.name == "OBJECT-TYPE"])
 }
The auto-complete still lists all Objects. However, upon choosing other than "OBJECT-TYPE", Xtext highlights it as an error.


Therefore, we are half way done, now need to make sure the auto-completion list is get filtered as well, apparently that is done separately in MibProposalProvider.xtend (This is under xtext generated ui project).

MibProposalProvider.xtend
class MibProposalProvider extends AbstractMibProposalProvider { 
 override completeObjectType_Imp(EObject model, Assignment assignment, ContentAssistContext context, ICompletionProposalAcceptor acceptor) {
  lookupCrossReference(assignment.terminal as CrossReference, context, acceptor, [(EObjectOrProxy as com.ravi.mib.xtext.mib.Object).name == "OBJECT-TYPE"])
 }
}

Now all works as it should be.

Few clarifications:
  • Should not make "OBJECT-TYPE" directly as keyword in Xtext as this is used as import. Sure, we can look for a way to overcome this i.e accepting as identifier or keyword based on context, however "OBJECT-TYPE" is rightfully just a macro and not a keyword in ASN.1 (Mib syntax is based on this). Therefore, I prefer to leave it as identifier.
  • I am not sure why auto-completion options for OidValue's parent works without the need to create similiar completeOidValue_parent() method. Hope someone can enlighten me on this.
  • Notice that in this round in method "scope_Identifier_imp", I have used "Scopes::scopeFor(...)" instead of  "new SimpleScope(Scopes::scopedElementsFor(...)". The latter gives us access to EObjectDescription (see also scopedElementsFor() with custom function argument to compute element name). However, for our case - no name customization is done, former is ideal.

Monday, 8 September 2014

Restricting auto-completion options - Scope wise

In MIB file, 2 values are required for oid value; parent object and object id. As for parent object name, editor should able to provide list of auto-completion based on previously defined object identifier plus objects that are imported from other MIB file.

The problem with this imported objects are not all are of object identifiers, some can be macro (OBJECT-TYPE) or some can be data types (NetworkAddress, IpAddress, Counter, Gauge etc). Hence its wrong to list all these as possible parent objects within the auto-completion list.


The screenshot illustrates the scenario. Therefore the challenge is to remove those options (macro and datatype) from the option list.

In order to do this, Xtext provides scope restriction. Since I have not (yet) look into cross reference with other MIB files, I will just be happy by hard-coding the exclusion list. This is temporary until I dive into MIB exports from dependency files.

While searching over net for the answer, Generally, I realized few things:

  • Unfortunately this is no longer declarative codes but need to be told imperatively - requires some 'coding'. We will use Xtend.
  • Xtend is something rather new in Xtext, therefore quite number of examples out there may use Java.
  • While Xtend simplify the expression, you may need to spend some time reading on Xtend for more effective coding!
Here is the code I wrote this achieve this:


MibScopeProvider.xtend
package com.ravi.mib.xtext.scoping

import org.eclipse.xtext.scoping.IScope
import com.ravi.mib.xtext.mib.OidValue
import org.eclipse.emf.ecore.EReference
import org.eclipse.xtext.scoping.Scopes
import com.ravi.mib.xtext.mib.Definition
import org.eclipse.xtext.EcoreUtil2
import org.eclipse.xtext.scoping.impl.SimpleScope
import java.util.HashSet

/**
 * This class contains custom scoping description.
 * 
 * see : http://www.eclipse.org/Xtext/documentation.html#scoping
 * on how and when to use it 
 *
 */
class MibScopeProvider extends org.eclipse.xtext.scoping.impl.AbstractDeclarativeScopeProvider {
 def IScope scope_OidValue_parent(OidValue value, EReference ref) {
  val excluded = new HashSet(#["NetworkAddress", "IpAddress", "Counter", "Gauge", "TimeTicks", "OBJECT-TYPE"])
  new SimpleScope(
   Scopes.scopedElementsFor(
    EcoreUtil2::getAllContentsOfType(value.eContainer.eContainer as Definition,
     com.ravi.mib.xtext.mib.Object).filter[!excluded.contains(it.name)]));
 }
}


MibScopeProvider.xtend is a xtend file pre-generated when Xtext project was created. I find it rather interesting that method name has lot to do with context where it will be called. "scope_OidValue_parent", "OidValue" is the related EObject for the rule and "parent" is EReference that we are interested.

Here is the complement rule to go with the code:

Mib.xtext
OidValue:
 '::=' '{' parent=[Object] oidnum=INT '}';

Here is the screenshot once this implemented:



Now for fun lets compare Xtend code vs Java code that generated from it, just to show arguably how much saving + readability Xtend can provide over Java.


Bash Terminal
dev@ravi-vm:~/workspace/com.ravi.mib.xtext$ wc ./src/com/ravi/mib/xtext/scoping/MibScopeProvider.xtend
  30   78 1030 ./src/com/ravi/mib/xtext/scoping/MibScopeProvider.xtend
dev@ravi-vm:~/workspace/com.ravi.mib.xtext$ wc ./xtend-gen/com/ravi/mib/xtext/scoping/MibScopeProvider.java
  52  153 2362 ./xtend-gen/com/ravi/mib/xtext/scoping/MibScopeProvider.java
Its almost half less coding!