Overview

When working with the Java related APIs in the NetBeans IDE we recommend making yourself familiar with some of the basic concepts which are used. The APIs are based directly on the APIs of the javac compiler. Some of the APIs are taken directly from javac. Some are added in order to integrate javac into the IDE properly and in order to supply functionality not provided by javac (e.g. modifying code). Being able to understand the basic of how javac works also helps.

Please note that there are some pitfalls when using these APIs. Make sure you know what these are and how to detect basic mistakes (both described later in the text) in order to not harm the performance of the IDE.

The essential packages a module writer will be interested in are:

• javax.lang.model.element, javax.lang.model.type, javax.lang.model.util. - For reading high level java language elements (classes, methods, fields, ...) and for working with types. These are the same APIs as those used for writing annotation processors (AKA JSR269).
• com.sun.source.tree, com.sun.source.util - For reading the syntactic structure of a java source i.e. AST (Abstract Syntax Trees)
• org.netbeans.java.source, org.netbeans.java.source.support - For integration with the IDE, additional utility methods and doing source code modifications.

The Javac phases

When compiling source files, javac works in phases. Knowing these phases can help you make your code more performant. When writing a task which looks into the source you should think first about what phase you need javac to be in. The less work required, the sooner your task will be run by the infrastructure. The important phases are:

• parse (syntactic analysis) - as result of this phase the AST is created and syntax errors are reported. You can think of this phase as a step enabling the usage of the com.sun.source.tree package. However, please note, that at this point the AST is NOT resolved, i.e. no type information is available. This mean that you can't tell at this point of what type an identifier is. For example you will be able to find out that there is a field of some name in the class, but will not be able to find out what class(es) can be assigned to the field. This may seem to be an uninteresting phase but, for example, some code completion code does not require type information, so keeping the javac in the parse phase speeds up the task's completion a bit.
• elements resolved - the type information is added for classes (not for local classes - innerclasses contained in bodies of methods) and their members, elements (see below) are attached to the tree. It is possible to get (resolved) return type of a method, get a type of a parameter, field, etc. The content of methods is not resolved at this phase, so no type information about the statements is available. This is the correct phase for those who want to work at the level of class, methods, fields, and other signatures but who are not interested in the code in method bodies. Yes, there are such modules. Think about stuff like JavaBeans support or a module working with Javadoc.
• resolved (aka attributed) - the type information is resolved for the whole source code. You can get type information for expressions inside the methods, etc. If you need to get all information about the code then this is the right phase for you.

Trees, Elements and Types

In order to efficiently model the Java 5.0 Language, three different hierarchies are defined:

• Trees com.sun.source.tree

Represents syntactic units in the source code, like method declaration, variable declaration, statement, etc. These packages describe the structure of a Java source files in detail. (See Javac Phases for the details about what information is included in the ASTs). These packages are useful if you want to deep dive into bodies of methods or if you require detailed info about the source (e.g. position of the elements etc). The package also contains visitor interfaces and scanner classes which you will need to implement and/or extend in order to walk through the AST.

• Elements javax.lang.model.element

This package describes a high-level model of the Java, which represent classes, interfaces, enums, annotations, methods, fields. constants etc. programming language. Element utility classes, such as filters, visitors and scanners are in the javax.lang.model.util package. The Elements class in this utility package contains several methods for working with elements.

• Types javax.lang.model.type

Represents particular types, like int, List, List<String>. With addition of generics in Java 5 the type system of the Java language got more complicated. Notice that one class (represented by an Element) can in fact represent n types. For example, MyClass<T> can be used as MyClass<String>, MyClass<JLabel>, etc. The type system is therefore defined in a separate package. This package handles primitive types, declared types, arrays, wild cards, type variables and so on. For utility classes e.g. filters, visitors and scanners look into javax.lang.model.util. For example, in the Types class in this package you can find interesting methods for answering questions like how a method of a generic class will look like in other type. ( e.g. List.<T>get(int i) will become String get( int i) in List<String>. There are many other interesting methods included in this class.

• Tokens (lexical information)

This is the lowest level info about the source. The information is produced by the java lexer. You can get to it using the NetBeans APIs described later. What it basically does is break up the source into tokens, each of which has its position and type assigned. You should not need this info very often. But in case you need it, it is there for your use.

All three hierarchies form the javac APIs. All the hierarchies are read only; if you want to make changes to the code you will have to go and consult the NetBeans API described later.

PITFALL - Don't try to implement the interfaces!

Even if all the APIs are done using interfaces. These interfaces are not there for users to implement them. Trying to provide your own implementation of a Tree, an Element or a Type and putting it back as a parameter to a method call to the APIs will very likely result into ClassCastException.

PITFALL - Don't relay on instanceof!

As there are several subinterfaces for various elements or trees (e.g. MethodTree, VariableTree, ExecutableElement, VariableElement, ... ) you may be tempted to test for it using instance of e.g. writing code like:

if( myTree instanceof VariableTree ) {
    ... do something with the variable ...
}

This is generally incorrect. Notice that it is not guaranteed that the interfaces that all or some of the interfaces are not implemented by the same class. Therefore results of instanceof operator may not be what you would expect.

Instead of using the instanceof operator rather call a getKind() method. This method returns an enum constant which describes the kind of the Tree, Element or Type correctly. See. ElementKind, TypeKind, Tree.Kind.

correct for of the code above would be

if( myTree.getKind() == Tree.Kind.VARIABLE ) {
    ... do something with the variable ...
}

Conversions between trees and elements and types...

Sometimes you need to go from element to a tree, or from a type to an element and vice versa.

• Going from a Tree to an Element use com.sun.source.util.Trees.getElement( TreePath ). Notice that you lose all the type information in case the class/method uses generics so you may be interested in instead using a Type rather than an Element.
• Going from a Tree to a Type use com.sun.source.util.Trees.getType( TreePath )
• Going from an Element to a Tree use one of the ( com.sun.source.util.Trees.getTree( ... ) methods.
• Going from an Element to a Type is easy; what you need to do is to call the asType() method on the Element.
• Going from a Type to an Element you can do this with DeclaredType instances by calling their asElement() method

PITFALL Methods for going from a Tree to an Element or a Type require TreePath

It is not enough to send the Tree as a parameter. There are basically several ways how to get a TreePath:

1. If you know the CompilationUnit call com.sun.source.util.Trees.getPath( CompilationUnitTree, Tree )
2. If you know the Element you can call com.sun.source.util.Trees.getPath( Element )
3. If you are going through the tree using a visitor or scanner you may rather want to subclass the com.sun.source.util.TreePathScanner class which will permit for getting the current path at any time using it's getCurrentPath() method.

Integration with the IDE

Now you know that there is the javac, how it works and what data it produces. The question is how the integration into the IDE is done. In the IDE there can be lots of modules interested in getting information about the sources. These modules have to "compete" for the information in a well defined way. This is where the javac phases come into play. A module writer can register tasks which will run after javac completes a given phase. (There is an enum for the phases JavaSource.Phase). We will talk about how to register a task later in this document. If there are more tasks for a given phase, which usually is the case, then tasks are sorted by priority. (Again there is an enum for the priorities JavaSource.Pritority).

You may wonder why you should create and register tasks. Why can't you just do some blocking calls into the infrastructure? To understand that think of javac as a scarce resource which has to be accessed by one thread only at a given time. In order to manage such resources you have to introduce some mechanism to access it e.g. locks, transactions or whatever. The NetBeans java infrastructure chose to do the access using a queue of tasks which are then run in defined order (as described above). This is very similar what Java platform does in the AWT. In AWT you also call InvokeLater(Runnable) where the runnable specifies what will be done when the task comes to run. The difference between AWT InvokeLater and a task in the Java infrastructure is that unlike in AWT the registered tasks run again when the source is modified. This approach permits the infrastructure to do some important things. E.g. cancelling currently running task and run some more important task. This may happen when the user starts typing in the editor or invokes code completion explicitly. At such events the user is more interested in seeing the result of the actions immediately rather than waiting for some low priority task to be finished.

PITFALL (big one) Don't hold objects from javac when your task finishes

This may seem strange at first look. But there is a good reason for not doing it. If you will remember a Tree, Type or Element outside of the task then the remembered object will hold all the javac created objects in memory. This may be a relatively large amount of data. Especially those classes which refer to many other classes have big closure of what has to be loaded and analysed in order to resolve the class correctly. Therefore remembering javac objects is very dangerous and may produce large memory leaks. Not only that after your task finishes the data may become useless very soon (when the file changes again). So you might be working on incorrect data which may later lead to further errors. There are ways around this problem (described later) e.g. using the ElementHandle.

List of packages/classes which should be used only inside of a task

com.sun.javadoc.*
com.sun.source.tree.* (except: com.sun.source.tree.Tree.Kind and com.sun.source.tree.TreeVisitor)
com.sun.source.util.* (except: com.sun.source.util.SimpleTreeVisitor, com.sun.source.util.TreeScanner, com.sun.source.util.TreePathScanner?)
javax.lang.model.element.* (except: ElementKind, ElementVisitor, AnnotationValueVisitor, Modifier, NestingKind)
javax.lang.model.type.* (except TypeKind and TypeVisitor)
javax.lang.model.util.Elements
javax.lang.model.util.Types
javax.tools.Diagnostics
org.netbeans.api.java.source.CompilationController
org.netbeans.api.java.source.CompilationInfo
org.netbeans.api.java.source.WorkingCopy
org.netbeans.api.java.source.ElementUtilities                           
org.netbeans.api.java.source.CommentUtilities  
org.netbeans.api.java.source.TreeUtilities
org.netbeans.api.java.source.Comment           
org.netbeans.api.java.source.TreeMaker

PITFALL Choose priority carefully

We've already explained why choosing the correct phase is important. The proper priority is important as well. Do not set the priority of your task too high. It is usually unnecessary and the higher priorities should remain reserved for tasks which have high demand on responsibility (e.g. code completion or coloring). Tasks which do things like putting annotations into the error stripe can usually wait a bit as it is fine to show these with some delay.

PITFALL Why tasks should be fast and really cancellable

The fact that the faster your tasks work the better for the IDE's performance is obvious. However, you should also try to make the task really cancellable. I.e. when the method cancel() is called on your task you should immediately stop the task. If you keep the task running you can considerably hurt the performance of other important tasks like editing, code completion, showing errors etc.

In many cases your task will need to traverse the AST. This is usually done using a visitor or scanner. In such cases you may want to look at the org.netbeans.java.source.support.CancellableTreeScanner/CancellableTreePathScanner, which will save you the work with checking for cancellation in your visitor.

Another smart thing to do when coding a module containing the tasks is to run your testing copy of the IDE with -J-Dorg.netbeans.api.java.source.JavaSource.reportSlowTasks=true which will report long running tasks and those tasks which do not react to cancel by printing them into the console.

Getting to the javac information = writing a java infrastructure task

After brief description of how the infrastructure works we need to look at writing a task in more detail. The interface you need to implement looks like:

public interface CancellableTask<P> {

    public void cancel();

    public void run( P parameter ) throws Exception;

}

We already mentioned what is the cancel() method good for. The more interesting and more important method obviously is the run() method. You may see that it takes a parameter of type P. P can be generally anything as the interface can be used in other context than the java infrastructure as well. However, when using it for implementation of the java related tasks the P can become: CompilationInfo, CompilationController or WorkingCopy. It depends in how you registered or started your task. Considering the ordering each of the parameter types adds some functionality you may use.

1. CompilationInfo - Used for read only tasks invoked by the infrastructure at given phase.
   • Permits for getting information about the source (Trees, Elements, Types)
   • Provides some utility classes for working with Trees, Elements and Types.
   • Knows FileObject and optionally the Document corresponding with the JavaSource
   • Knows the the errors (Diagnostics) the javac run into
   • Knows the token hierarchy
   • Knows the phase the compiler is in
2. CompilationController - Used for read-only tasks which should be invoked on user's action. Permits for moving Javac into given phase.
   • Can do everything what the CompilationInfo does
   • Can move javac to given phase
3. WorkingCopy - Used for tasks which do modifications to the source
   • Does everything what the CompilationController does
   • Permits for doing modifications to the source

Registering/running tasks

The last thing to do with a correctly implemented task is to let it really run. There are two basic ways how to do it. First, register it into the infrastructure so that it will be run automatically when the source gets modified. Second, running it directly.

Use the former option if:

• You need to react to changes of document in the editor
• You need to react to changes in caret position in the editor
• You want to do something with files which are collected in some existing lookup
• You need to run a task on arbitrary file/set of files every time a file changes

Use the later option:

• If the task has to be run after invocation from menu, through shortcut etc. (For how to do it look at How do I get into Javac context)

Registering tasks using the factories

There are support classes for the most used cases of repeatingly running tasks. The classes are in the org.netbeans.java.source.support package. Generally you need to subclass the factory of your choice (see later) and implement the abstract methods or override some methods. You will need to implement the getPhase() and getPriority() methods and return an enum constant of your choice in order to indicate at which javac phase and with which priority the tasks created by this factory should run. (See above for discussion about phases and priorities). The most important method to implement is createTask( FileObject ), which has to return the task to be run.

Once you have your factory implemented you will need to register it into the global Lookup. To do so create a folder called META-INF.services in your project and put a file called org.netbeans.java.source.JavaSourceTaskFactory (note: has this changed to org.netbeans.api.java.source.JavaSourceTaskFactory?) in the folder. In this file list the fully qualified names (one per line) of all factories you want to register. So if you want to register two factories the file content would look like:

foo.bar.my.java.tasks.MyEdtiorAwareTaskFactory
foo.bar.my.java.tasks.MyCaretAwareTaskFactory

• Reacting to changes in the editor
  Is easy. Subclass EditorAwareJavaSourceTaskFactory and implement all abstract classes. Your class could look like:

public class JavaSourceTaskFactoryImpl extends EditorAwareJavaSourceTaskFactory {
    
    public CancellableTask<CompilationInfo> createTask(FileObject file) {
        return new MyTask();
    }
    
    public Priority getPriority() {
        return Priority.LOW;
    }
    
    public Phase getPhase() {
        return Phase.RESOLVED;
    }
    
}

The run( CompilationInfo ) of your task will then be run every time a file changes in the editor and the compiler gets into the state where all identifiers are resolved.

• Reacting to a caret position change
  Do the same thing as in the previous paragraph; just make sure you extend CarretAwareJavaSourceTaskFactory

• Tracking changes of files in a lookup
  This might sound like a strange thing to do, but in NetBeans there are often lookups available which contain interesting files. For instance the Java navigator component can be written this way. You extend the LookupBasedJavaSourceTaskFactory you call the setLookup(Lookup) method on it pointing it to the proper lookup. The rest is identical to steps described in the first bullet point.
  Now when any of the files in the lookup changes the task you created for it will run. Should you want to ignore some of the files override the getFiles() method and do the filtering in it.

• Reacting to changes in an arbitrary set of files
  If you think you need this, think again. Maybe you don't. If you still think you do subclass the JavaSourceTaskfactorySupport class. Implement the getFiles method so that it will return the FileObjects you are interested. If the set of files is not fixed call fireChangeEvent() every time the set of files changes. Follow the first bullet point to implement all other abstract methods.

ElementHandle

As was mentioned above, it is not possible to hold Elements, TypeMirrors and Trees and to compare instances got from different javac instances.

To solve this limitation, ElementHandle was introduced. The ElementHandle allows to pass an Element from one javac instance to another. In the first instance of javac, create the handle using ElementHandle.create(Element) method. Keep the resulting handle, but do not hold the element itself. In the other instance of javac, use ElementHandle.resolve(CompilationInfo) method to resolve the ElementHandle back to the Element.

TreePathHandle

As was mentioned above, it is not possible to hold Elements, TypeMirrors and Trees and to compare instances got from different javac instances.

To solve this limitation, ElementHandle and TreePathHandlewas introduced. The TreePathHandle allows to pass an TreePath from one javac instance to another. In the first instance of javac, create the handle using TreePathHandle.create(Element, CompilationInfo) method. Keep the resulting handle, but do not hold the tree or tree path itself. In the other instance of javac, use TreePathHandle.resolve(CompilationInfo) method to resolve the TreePathHandle back to the TreePath.

Source Files vs. Class Files

Parsing a file from the source code is time and memory consuming task, compared to simple loading of a class file. So, it is important for all the authors of a code in the IDE to know if their code requires parsing a source file or if it is sufficient to use data from a class file.

Another significant difference between these two approaches is that the Trees are available only if the file has been compiled from the source file. The Elements load from a class file are supposed to be "identical" to elements got by parsing the corresponding source file.

The rule of thumb here is that if you need only classes, their methods, constructors, fields and inner classes and their annotations, loading the data from the class file is sufficient for you. If you need access of the method content (statements), etc., you need to get data by parsing the source code.

Class Files vs. Signature Files

Although the previous text spoke about loading class files, the IDE does not use true class files much. When a file is compiled by the IDE, the result is not a class file, but rather a "signature" file. These files contain specification of classes and signatures of their members. The main differences between class files and signature files are:

• the signature files do not contain method bodies
• the signature files do contain annotations with RetentionPolicy.SOURCE
• the signature files do contain parameter names (which are stored only as part of the debug information into the class files)

The format of the signature files is subject of change at any time, as it is used only to create the IDE's caches.

Note: for most purposes, it is not important if an Element has been load from the class file or from the signature file.

How to setup a module which uses the Java related APIs

TBD: Describe necessary module dependencies

How do I Get into the Javac Context for a File?

First of all, think about what are you trying to do. There are different ways to enter the javac context for different usecases. You may go and consult Registering/Running tasks first. Then if you decide that this is not the way you may continue reading here.

Find answers to these questions:

1. When the code is supposed to run
   1. On explicit user action, like when user invokes an action through menu, or when the user invokes the code completion.
   2. "When the file is parsed by the infrastructure" - like the coloring, editor hints, etc.
2. What is the scope of the code
   1. one file - usually the one visible in the editor - like coloring, code completion, editor hints
   2. many files (eg. whole project) - like find usages
3. Does the code require Trees?
4. Will the code make any modifications into the source code (through the TreeMaker API)?

Please note that not all combinations of answers to the above questions make sense.

In all cases, you will need to create a CancellableTask, which will do the work.

	Read-only access, one file	Read-only access, more files	Write access, one file	Write access, more files
Trees, on user action	I	II	III	IV
no trees, on user action	V	V	x (no write access without trees)	x (no write access without trees)
Trees, automatically	VI	x	x	x
no Trees, automatically	x	x	x	x

I.: use JavaSource.forFileObject(/given file object/).runUserActionTask(CancellableTask<CompilationController>). Learn more on CompilationController usage ?below?.

II.: use JavaSource.create(ClasspathInfo, FileObject...).runUserActionTask(CancellableTask<CompilationController>). Learn more on CompilationController usage ?below?. Please see javadoc for the JavaSource.create method for more detailed information on how this works.

III.: use JavaSource.forFileObject(/given file object/).runModificationTask(CancellableTask<WorkingCopy>). Learn more on WorkingCopy usage ?below?. The runModificationTask method returns ModificationResult, which represents "textual" diff for changes the CancellableTask did. To commit these changes, perform ModificationResult.commit().

IV.: use JavaSource.create(ClasspathInfo, FileObject...).runModificationTask(CancellableTask<WorkingCopy>). Learn more on WorkingCopy usage ?below?. The runModificationTask method returns ModificationResult, which represents "textual" diff for changes the CancellableTask did. To commit these changes, perform ModificationResult.commit(). Please see javadoc for the JavaSource.create method for more detailed information on how this works.

V.: use JavaSource.create(ClasspathInfo)

How do I Get All Methods/Fields/Constructors of a Class?

• First, you need to be in a Javac context, see previous section for more information.
• Then, you need to find javax.lang.model.element.TypeElement you want to analyze. See com.sun.source.tree.Trees.getElement(TreePath) and javax.lang.model.util.Elements.getTypeElement(String). You can get Trees and Elements from org.netbeans.api.java.source.CompilationInfo.
• Finally, use Element.getEnclosedElements() to find out the elements enclosed by the class - for classes, this returns all members (methods, fields and inner classes) of the class. You can then use ElementFilter to filter out specific kind of member: methods, constructors, fields and inner classes.

Example:

protected void performAction(Node[] activatedNodes) {
    DataObject dataObject = (DataObject) activatedNodes[0].getLookup().lookup(DataObject.class);
    JavaSource js = JavaSource.forFileObject(dataObject.getPrimaryFile());

    try {
        js.runUserActionTask(new CancellableTask<CompilationController>() {
            public void cancel() {}
            public void run(CompilationController parameter) throws IOException {
                parameter.toPhase(Phase.ELEMENTS_RESOLVED);
                new MemberVisitor(parameter).scan(parameter.getCompilationUnit(), null);
            }
        }, true);
    } 
    catch (IOException e) {
        Logger.getLogger("global").log(Level.SEVERE, e.getMessage(), e);
    }
}

private static class MemberVisitor extends TreePathScanner<Void, Void> {

    private CompilationInfo info;

    public MemberVisitor(CompilationInfo info) {
        this.info = info;
    }

    @Override
    public Void visitClass(ClassTree t, Void v) {
        Element el = info.getTrees().getElement(getCurrentPath());


        if (el == null) {
            System.err.println("Cannot resolve class!");
        } 
        else {
            TypeElement te = (TypeElement) el;
            System.err.println("Resolved class: " + te.getQualifiedName().toString());
            //XXX: only as an example, uses toString on element, which should be used only for debugging
            System.err.println("enclosed methods: " + ElementFilter.methodsIn(te.getEnclosedElements()));
            System.err.println("enclosed types: " + ElementFilter.typesIn(te.getEnclosedElements()));
        }
        return null;
    }

}

How do I modify a source file?

Most modifications are done through the API. Direct document changes are not recommended. Editing source through the API has many advantages, for instance it respects formatting settings.

This part will show you typical steps to make a modification to your source. There can be found different usecases, but this is the most common:

• Find the JavaSource you want to work with,
• create a task that contains code for source modification,
• post the task to the JavaSource and commit the changes at the end.

TODO: link to example, describe what the example does.

Find the JavaSource

There are more ways to do it. For our demonstration, we use a straightforward solution, often used in tests. We omit the details of getting fileObject and we expect successful behaviour of called methods.

    File tutorialFile = getFile(getSourceDir(), "/org/netbeans/test/codegen/Tutorial1.java");
    JavaSource tutorialSource = JavaSource.forFileObject(FileUtil.toFileObject(tutorialFile));

The JavaSource represents the file Tutorial1.java in package org.netbeans.test.codegen.

Create a 'modify' task

The task has to be cancellable. CancellableTask is a parameterized interface with type parameter WorkingCopy. This type is requested in the next step. This next code snippet shows how to create an anonymous CancellableTask:

    CancellableTask task = new CancellableTask<WorkingCopy>() {
        ...
    }

The interface contains two method, perhaps no surprise here:

• run() - contains code for modifying our javaSource,
• cancel() - defines what to do when cancel() is processed.

    CancellableTask task = new CancellableTask<WorkingCopy>() {

        public void run(WorkingCopy workingCopy) throws Exception {
            ... our modification code
        }

        public void cancel() {
            ... cancel code
        }

    };

The run method contains all staff describing modification and we will dive in to the details later.

Post the task to process and commit the changes

Because we want to modify the source, we have to use runModificationTask (see its javadoc). At the end, we have to commit changes to propagate all the work to the source file - our Tutorial1.java file. This can fail, so ensure you correctly handle exceptions. The method runModificationTask() returns the modification result. This class contains all the prepared changes which haven't been propagated yet. This is good especially when someone wants to review the details of modification and decide about propagating changes to the source on the basis of result. For our demonstration, we will omit it. When result is collected, we have to call the commit() method to propagate the changes to the source code:

    ModificationResult result = tutorialSource.runModificationTask(task);
    result.commit();

Implementing run() method

The run() method is place where we you API for modifications. We decided to add java.io.Externalizable interface the class declaration.

Original source looks like:

    package org.netbeans.test.codegen;

    public class Tutorial1 {
    }

At the end, we want to see source like:

    package org.netbeans.test.codegen;

    import java.io.Externalizable;

    public class Tutorial1 implements Externalizable {
    }

In short, this can be done with this code:

    public void run(WorkingCopy workingCopy) throws IOException {

        workingCopy.toPhase(Phase.RESOLVED); // is it neccessary?

        CompilationUnitTree cut = workingCopy.getCompilationUnit();
        TreeMaker make = workingCopy.getTreeMaker();

        for (Tree typeDecl : cut.getTypeDecls()) {
            if (Tree.Kind.CLASS == typeDecl.getKind()) {
                ClassTree clazz = (ClassTree) typeDecl;
                ExpressionTree implementsClause = make.Identifier("Externalizable");

                implementsClause = make.Identifier("java.io.Externalizable");
                        
                TypeElement element = workingCopy.getElements().getTypeElement("java.io.Externalizable");
                implementsClause = make.QualIdent(element);
                        
                ClassTree modifiedClazz = make.addClassImplementsClause(clazz, implementsClause);
                workingCopy.rewrite(clazz, modifiedClazz);
            } // end if
        } // end for
    } // end run

Here is steps description:

• workingCopy.toPhase(Phase.RESOLVED); -- Resolves symbols for provided java source.
• CompilationUnitTree cut = workingCopy.getCompilationUnit(); -- Instance represents one java source file, exactly as defined in JLS, §7.3 Compilation Units.
• TreeMaker make = workingCopy.getTreeMaker(); -- Get the tree maker, the core class used for making modifications. It allows to add new members to class, modify statements, etc.
• for (Tree typeDecl : cut.getTypeDecls()) { ... } -- Go through all top level declarations (JLS §7.6).
• if (Tree.Kind.CLASS == typeDecl.getKind()) { ...} -- Ensure about type - not neccessary here, when we omit that Annotation Type can be also declared here. This is important and you will see it perhaps on other places too -- always, you have to use Kind for checking instance! instanceof operator shouldn't be used for such a test.
• ClassTree clazz = (ClassTree) typeDecl;
• Create identifier:
  • ExpressionTree implementsClause = make.Identifier("Externalizable"); -- Simpliest, but not sufficient solution: Add the plain identifier. It generates source as you can see below, but when import is not available, identifier is not resolved and class will not compile.

    public class Tutorial1 implements Externalizable {
    }

• • ExpressionTree implementsClause = make.Identifier("java.io.Externalizable"); -- We can solve described problem with specifying fully-qualified name. We can create again identifier tree. (Bear in mind, that you will never get such an identifier - meant dot separated - from the compiler staff. Note: Should we consider it as incorrect usage?) The result will be compilable, see code below. The disadvantage is fully-qualified name in declaration.

    public class Tutorial1 implements java.io.Externalizable {
    }

• • Last, and perhaps the most often used solution is to add plain identifier to type declaration and correct import statement to compilation unit. It can be done by following statements: TypeElement element = workingCopy.getElements().getTypeElement("java.io.Externalizable"); -- You will get resolved element. You should check, that element is available. Then, make QualIdent tree: implementsClause = make.QualIdent(element); -- The QualIdent will be recognized during source code modification and engine will decide (in accordance with options), how to correctly generate. When using default settings, import for your class will be added and simple name will be used in implements clause:

    import java.io.Externalizable;

    public class Tutorial1 implements Externalizable {
    }

• ClassTree modifiedClazz = make.addClassImplementsClause(clazz, implementsClause); -- Use tree maker method to add the interface identifier to the 'implements' clause. Bear in mind that this operation just put it to the tree, not to the source file. Because nodes in tree are immutable, method returns the same class type as provided in first parameter, in our case ClassTree. In other words, if a method takes ClassTree parameter, it will return another class tree, which contains provided modification.
• workingCopy.rewrite(clazz, modifiedClazz); -- Replace the original node with the new one. It adds the change to the list of changes, later used for making source modification.

Adding method with body

Next example show more complex task. Adding method to type declaration. The steps described above are the same, we just implement run() method of CancellableTask. Here is the code we want to add to class declaration:

    public void writeExternal(final ObjectOutput arg0) throws IOException {
        throw new UnsupportedOperationException("Not supported yet.");
    }

We have to prepare all elements belonging to method. First of all, prepare modifiers for the method. We use TreeMaker instance make again.

    public void run(WorkingCopy workingCopy) throws IOException {
        ...
        // create method modifier: public and no annotation
        ModifiersTree methodModifiers = make.Modifiers(
            Collections.<Modifier>singleton(Modifier.PUBLIC),
            Collections.<AnnotationTree>emptyList()
        );
        ...
    }

Next step is preparing method parameter arg0 of type ObjectOutput and modifier final:

                     
    public void run(WorkingCopy workingCopy) throws IOException {
        ...
        // create parameter:
        // final ObjectOutput arg0
        VariableTree parameter = make.Variable(
            make.Modifiers(
                Collections.<Modifier>singleton(Modifier.FINAL),
                Collections.<AnnotationTree>emptyList()
            ),
            "arg0", // name
            make.Identifier("Object"), // parameter type
            null // initializer - does not make sense in parameters.
        );
        ...
    }

Method throws exception, prepare exception identifier IOException. It is the same when we prepared interface for implements clause.

    public void run(WorkingCopy workingCopy) throws IOException {
        ...
        // prepare simple name to throws clause:
        // 'throws IOException' and its import will be added (if it is not available yet)
        TypeElement element = workingCopy.getElements().getTypeElement("java.io.IOException");
        ExpressionTree throwsClause = make.QualIdent(element);
        ...
    }

We have everything, what we need for method creation. Make method:

    public void run(WorkingCopy workingCopy) throws IOException {
        ...
        // create method.
        MethodTree newMethod = make.Method(
            methodModifiers, // public
            "writeExternal", // writeExternal
            make.PrimitiveType(TypeKind.VOID), // return type "void"
            Collections.<TypeParameterTree>emptyList(), // type parameters - none
            Collections.<VariableTree>singletonList(parameter), // final ObjectOutput arg0
            Collections.<ExpressionTree>singletonList(throwsClause), // throws 
            "{ throw new UnsupportedOperationException(\"Not supported yet.\") }", // body text
            null // default value - not applicable here, used by annotations
        );
        ...
    }

In the example above, we used the most often used factory method for source code method creation. It contains string for its body. You can add it as plain syntax correct text and engine will do imports and formatting stuff for you. There is also second method, which allows to add the body as a block:

    public void run(WorkingCopy workingCopy) throws IOException {
        ...
        // create method.
        MethodTree newMethod = make.Method(
            methodModifiers, // public
            "writeExternal", // writeExternal
            make.PrimitiveType(TypeKind.VOID), // return type "void"
            Collections.<TypeParameterTree>emptyList(), // type parameters - none
            Collections.<VariableTree>singletonList(parameter), // final ObjectOutput arg0
            Collections.<ExpressionTree>singletonList(throwsClause), // throws 
            make.Block(Collections.<StatementTree>emptyList(), false), // empty statement block
            null // default value - not applicable here, used by annotations
        );
        ...
    }

Example creates method with empty body.

At the end, do not forget to add it to type declaration and register change on working copy:

        // and in the same way as interface was added to implements clause,
        // add feature to the class as its member:
        ClassTree modifiedClazz = make.addClassMember(clazz, newMethod);
        workingCopy.rewrite(clazz, modifiedClazz);
        ...
    }

Do you want to see it in a practice? Open the java/source project, go to unit test packages, then org.netbeans.api.java.source.gen package, open file TutorialTest.java and run it in IDE. You can experiment with it little bit.

TreeMaker Difficulties, Questions and Answers

Question: How to rewrite Java/MDR modification code to new API?

Answer: Most of the actions can be replaced by the new method. There are several conceptual differences: The most important and visible is immutability of the new trees, so you cannot modify original model elements (in Java/MDR speech), you have to always create new tree (in Retouche speech). Here are couple of examples:

Question: I need to make textual replace of the tags from the template with some values in method body

Answer: Currently there is no direct way how to do that. We could consider to add method to obtain body as a text to TreeMaker class, but bear in mind that such a method does not modify source, so it is not correct place. For the time being, you can workaround it by following code:

(Code will not work, there are still many bugs.)

    ...
    TreeMaker make = workingCopy.getTreeMaker();
    ...
    MethodTree method = ...;
    BlockTree body = method.getBody();

    // get SourcePositions instance for your working copy and
    // fetch out start and end position.
    SourcePositions sp = workingCopy.getTrees().getSourcePositions();
    int start = (int) sp.getStartPosition(cut, body);
    int end = (int) sp.getEndPosition(cut, body);
    // get body text from source text
    String bodyText = workingCopy.getText().substring(start, end);
    MethodTree modified = make.Method(
        method.getModifiers(), // copy original values
        method.getName(),
        method.getReturnType(),
        method.getTypeParameters(),
        method.getParameters(),
        method.getThrows(),
        bodyText.replace("{0}", "-tag-replace-"), // replace body with the new text
        null // not applicable here
    );
    // rewrite the original modifiers with the new one:
    workingCopy.rewrite(method, modified);
    ...

Question: How do I create a constructor? Passing TypeKind.VOID as the return type to TreeMaker.Method() generates a method with a void return type.

Answer: by passing "<init>" as the method name and null as the return type. See also issue 88697.

Question: Create an Import and add it to a class

Retouche:

TreeMaker make;
CompilationUnitTree cut = ...;
CompilationUnitTree copy = make.addCompUnitImport(
    cut, 
    make.Import(make.Identifier("java.io.IOException"), false)
);
workingCopy.rewrite(node, copy);

Note: There is a management tool which allows to handle imports automatically. Use make.QualIdent to create expression tree - import will be handled for it, if it does not exist yet.

Question: Create that some imports are not already there

This is some kind of workaround. Users should use automatic import management (QualIdentTree) in the most cases. For those who request manual handling, use similar scenario in retouche:

CompilationUnitTree cut = ...;
List<ImportTree> imports = cut.getImports();
for (ImportTree dovoz : imports) [
   if ("whateverYouWant".equals(dovoz.getQualifiedIndetifier().toString())) {
       found = true; break;
   }

Note: It is not recommended to make your own import management, instead of, use the automatic one. Use your own solution only if automatic one does not fit to your needs!

Question: Create a method

It is very similar to other creations. All the stuff is done in TreeMaker. Here is a short example: Retouche:

// create modifiers for parameters (no modifier present) and create annotations. (again empty list)
ModifiersTree parMods = make.Modifiers(Collections.<Modifier>emptySet(), Collections.<AnnotationTree>emptyList());
// make a variable trees - representing parameters
VariableTree par1 = make.Variable(parMods, "a", make.PrimitiveType(TypeKind.INT), null);
VariableTree par2 = make.Variable(parMods, "b", make.PrimitiveType(TypeKind.FLOAT), null);
List<VariableTree> parList = new ArrayList<VariableTree>(2);
parList.add(par1);
parList.add(par2);

// now, start the method creation
MethodTree newMethod = make.Method(
            make.Modifiers(Collections.singleton(Modifier.PUBLIC)), // modifiers and annotations
            "newlyCreatedMethod", // name
            make.PrimitiveType(TypeKind.VOID), // return type
            Collections.<TypeParameterTree>emptyList(), // type parameters for parameters
            parList, // parameters
            Collections.singletonList(make.Identifier("java.io.IOException")), // throws 
            make.Block(Collections.EMPTY_LIST, false), // empty statement block
            null // default value - not applicable here, used by annotations
        );

Question: Add a method to a class

Consider you have created method and want to add it to a superclass:

Java/MDR:

Class clazz = ...;
Method method = ...;
clazz.getContents().add(method);

Retouche:

TreeMaker make = ...;
ClassTree clazz = ...;
MethodTree method = make.Method(...);
ClassTree copy = make.addClassMember(clazz, method);
workingCopy.rewrite(clazz, copy);

Question: Set a super class

Java/MDR:

Class clazz = ...;
clazz.setSuperClassName("motherClassName");

Retouche:

ClassTree clazz = ...;
ClassTree copy = clazz.setExtends(clazz, make.Identifier("MotherClassName");
workingCopy.rewrite(clazz, copy);

Question: Add a parameter to a constructor

Java/MDR:

Constructor construct = tgtClass.getConstructor(new ArrayList(), false);
Parameter param = pkg.getParameter().createParameter(
        "theRef", // NOI18N
        Collections.EMPTY_LIST, // annotations
        false, // is final
        getTypeRef(pkg, mbean.getWrappedClassName()), // typename
        0, // dimCount
        false);
construct.getParameters().add(param);

Retouche:

TreeMaker make = ...;
ClassTree clazz = ...;
MethodTree constr = ...;
VariableTree var = make.Variable(make.Modifiers(Collections.<Modifier>emptySet()), "theRef", make.Identifier("someType"), null);
MethodTree copy = make.addMethodParameter(constr, var);
workingCopy.rewrite(constr, copy);

Question: Add a field to a class

Java/MDR:

Class clazz = ...;
Field refField = pkg.getField().createField("theRef", Collections.EMPTY_LIST, Modifier.PRIVATE, null, ...);
clazz.getFeatures().add(0, refField);

Retouche:

TreeMaker make = ...;
ClassTree clazz = ...;
VariableTree var = make.Variable(make.Modifiers(Modifier.PUBLIC), "theRef", make.Identifier("someType", null);
ClassTree clazzCopy = make.insertClassMember(0, var);
workingCopy.rewrite(clazz, clazzCopy);

Note: All fields, local variables and parameters is represented by VariableTree in Jsr199.

Question: Remove constructor's 'throws' clause

Java/MDR:

Constructor construct = ...;
construct.getExceptionNames().clear();

Retouche:

TreeMaker make = ...;
MethodTree method = ...;
MethodTree modified = make.Method( // copy original values
    method.getModifiers(),
    method.getName(),
    method.getReturnType(),
    method.getTypeParameters(),
    method.getParameters(),
    Collections.<ExpressionTree>emptyList(), // use empty list instead of orig. value
    method.getBody(),
    null // not applicable here
);
workingCopy.rewrite(method, modified);

For exact 'throws' item removal, you can use methods make.removeMethodThrows(...) in TreeMaker class.

Question: Clear body text

Java/MDR:

Method method = ...;
method.setBodyText("");

Retouche:

TreeMaker make = ...;
MethodTree method = ..;
BlockTree emptyBlock = make.Block(Collections.<StatementTree>emptyList(), false);
workingCopy.rewrite(method.getBody(), emptyBlock);

Question: Type object

Question: Access to the String representation of a method body

There is not any direct support for such a functionality. You have to obtain positions and then cut the string from the source. We consider about adding such a method somewhere to the API. (Currently no suitable places has been found.)

Java/MDR:

    String bodyText = method.getBodyText();

Retouche:

    ...
    TreeMaker make = workingCopy.getTreeMaker();
    ...
    MethodTree method = ...;
    BlockTree body = method.getBody();

    // get SourcePositions instance for your working copy and
    // fetch out start and end position.
    SourcePositions sp = workingCopy.getTrees().getSourcePositions();
    int start = (int) sp.getStartPosition(cut, body);
    int end = (int) sp.getEndPosition(cut, body);
    // get body text from source text
    String bodyText = workingCopy.getText().substring(start, end);

Question: Contextualize the Java classes search

Question: Add a set of Exceptions to a method

Java/MDR:

method.getExceptionNames().addAll(exceptions);

Retouche:

MethodTree node = ...; // original MethodTree node to modify
MethodTree copy = make.addMethodThrows(node, make.Identifier("IOException"));
copy = make.addMethodThrows(copy, make.Identifier("FileNotFoundException"));
workingCopy.rewrite(node, copy);

Short comment about functionality is available below in next question.

Question: How to make the class implements an interface

Java/MDR:

JavaModelPackage pkg = (JavaModelPackage) tgtClass.refImmediatePackage();
tgtClass.getInterfaceNames().add(pkg.getMultipartId().createMultipartId(
            "MBeanRegistration", // NOI18N
            null,
            Collections.EMPTY_LIST));

Retouche:

workingCopy.toPhase(Phase.RESOLVED);
TreeMaker make = workingCopy.getTreeMaker();
ClassTree clazz = ...; // obtain class somewhere
ClassTree copy = class.addImplementsClause(make.Identifier("MBeanRegistration"));
workingCopy.rewrite(class, copy);

Java/MDR:

method.getExceptionNames().addAll(exceptions);

Retouche:

MethodTree method = ...;
List<ExpressionTree> listCopy = new ArrayList<ExpressionTree>(method.getThrows());
listCopy.add(exceptions);
MethodTree copy = make.Method(
       method.getModifiers(), 
       method.getName(), 
       method.getReturnType(), 
       method.getTypeParameters(), 
       method.getParameters(), 
       listCopy, 
       method.getBody(),
       null
);
workingCopy.rewrite(method, copy);

From the example above, it is obvious that this solution is not straightforward enough. You have also another chance how to do it:

Collection<IdentifierTree> exceptions = ...;
MethodTree method = ...;
MethodTree copy = make.addMethodThrows(method, make.Identifier("IOException"));
copy = make.addMethodThrows(copy, make.Identifier("FileNotFoundException"));
copy = make.addMethodThrows(copy, make.Identifier("IllegalArgumentException));
... etc.
workingCopy.rewrite(method, copy);

Bear in mind that in every next 'add' call, you have to use created copy instead of the original method MethodTree! (See the first parameter of addMethodThrows invocation.) This solution is shorter and simple to write, but it has minor performance impact when adding many items .It is up to user if he uses first or second solution, the result is the same. It is obvious that if you want to add just one item, you will use provided addMethodThrows method.

Note: Users can be confused when add multiple 'extends' clause to interface. Extends clause in interface is represented by 'implements' clause in class.

Question: How do I create a field with an initial value if the initial value is not a primitive, for example something like the following?

MyClass field = Something.getMyClass();

Answer: Obviously you have to create appropriate tree, in the example above method invocation with member select tree inside.

Such a code might look like:

...
CompilationUnitTree cut = workingCopy.getCompilationUnit();
TreeMaker make = workingCopy.getTreeMaker();
ClassTree clazz = (ClassTree) cut.getTypeDecls().get(0);
VariableTree var = make.Variable(make.Modifiers(
    Collections.<Modifier>emptySet(), Collections.<AnnotationTree>emptyList()),
    "myField",
    make.Identifier("MyClass"),
    make.MethodInvocation(
    Collections.<ExpressionTree>emptyList(), 
    make.MemberSelect(
        make.Identifier("Something"), "getMyClass"),
        Collections.<ExpressionTree>emptyList()
    )
);
ClassTree copy = make.addClassMember(clazz, var);
workingCopy.rewrite(clazz, copy);
...

Question: How to access and update the Javadoc of an existing class.

Currently totally broken, there are issues reported and will be addressed in M7: (#89873, #90302, #92325) There are methods in CommentHandlerService, they will be perhaps reused.

Tip: If you are not familiar with trees enough, write sample source code, run user action task against the source code and dump the tree to some readable form. That allows you to learn how expressions are represented in tree.