5. Transform Development Quick-Start

The following sections will give you a quick-start tutorial on how to develop transforms.

5.1 – Creating a Transform Package

Developing transforms is now easier than ever. If you want to create a whole bunch of transforms or if you wish to take advantage of canari install-package then you’ll want to create a transform package. Otherwise, you’ll have to manually install and configure your local transform in the Maltego UI. We’ll just go ahead and create a transform package called mypackage because I have a good feeling you’ll be really eager to create a whole bunch of transforms:

1
2
3
4
5
6
7
8
9
10
$ canari create-package mypackage
creating skeleton in mypackage
creating file setup.py…
creating file README.md…
creating file src/mypackage/transforms/common/entities.py…
creating file src/mypackage/transforms/helloworld.py…
creating file src/mypackage/__init__.py…
creating file src/mypackage/transforms/__init__.py…
creating file src/mypackage/transforms/common/__init__.py…
done!

You’ll notice that a simple skeleton project was generated, with a helloworld transform to get you started. You can test the helloworld transform module by running canari debug-transform like so:

1
2
3
4
5
6
7
8
9
10
11
12
$ canari debug-transform mypackage.transforms.helloworld Phil
%50
D:This was pointless!
%100
`- MaltegoTransformResponseMessage:
  `- Entities:
    `- Entity:  {‘Type’: ‘test.MyTestEntity’}
      `- Value: Hello Phil!
      `- Weight: 1
      `- AdditionalFields:
        `- Field: 2 {‘DisplayName’: ‘Field 1′, ‘Name’: ‘test.field1′, ‘MatchingRule’: ‘strict’}
        `- Field: test {‘DisplayName’: ‘Field N’, ‘Name’: ‘test.fieldN’, ‘MatchingRule’: ‘strict’}

5.2 – Developing a Transform

Let’s take a look at an abbreviated version of src/mypackage/transforms/helloworld.py, from our example above, to see how this transform was put together.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#!/usr/bin/env python

from canari.maltego.entities import Person
from canari.maltego.utils import debug, progress
from canari.framework import configure #, superuser
from common.entities import MypackageEntity

# …
#@superuser
@configure(
    label=‘To MypackageEntity [Hello World]‘,
    description=‘Returns a MyPackageEntity entity with the phrase "Hello Word!"’,
    uuids=[ ‘mypackage.v2.MyPackageEntityToPhrase_HelloWorld’ ],
    inputs=[ ( ‘MyPackageEntity’, Person ) ],
    debug=True
)
def dotransform(request, response):
    # Report transform progress
    progress(50)
    # Send a debugging message to the Maltego UI console
    debug(‘This was pointless!’)

    # Create MyPackageEntity entity with value set to ‘Hello <request.value>!’
    e = MypackageEntity(‘Hello %s!’ % request.value)

    # Setting field values on the entity
    e.field1 = 2
    e.fieldN = ‘test’

    # Update progress
    progress(100)

    # Add entity to response object
    response += e

    # Return response for visualization
    return response


def onterminate():
    debug(‘Caught signal… exiting.’)
    exit(0)

Right away, you notice that there are a whole bunch of decorators (or annotations) and two functions (dotransform and onterminate). So what does this all mean and how does it work? Let’s focus on the meat, shall we?

The dotransform function is the transform’s entry point, this is where all the fun stuff happens. This transform isn’t particularly fun, but it serves as a good example of what typically happens in a Canari transform. dotransform takes two arguments, request and response. The request object contains the data passed by Maltego to the local transform and is parsed and stored into the following properties:

  • value: a string containing the value of the input entity.
  • fields: a dictionary of entity field names and their respective values of the input entity.
  • params: a list of any additional command-line arguments to be passed to the transform.

The response object is what our data mining logic will populate with entities and it is of type MaltegoTransformResponseMessage. The response object is very neat in the sense that it can do magical things with data. With simple arithematic operations (+=, -=, +, -), one can add/remove entities or Maltego UI messages. You’ll probably want to use the += or -= operators because - and + create a new MaltegoTransformResponseMessage and that can be costly. Let’s take a look at how it works in the transform above:

1
2
3
4
5
# …
    e = MypackageEntity(‘Hello %s!’ % request.value)
# …
    response += e
# …

The first line of code, creates a new MypackageEntity object is created with a value ‘Hello <request.value>!’. The second line of code adds the newly created object, e, to the response object. If we serialize the object into XML we’d see the following (spaced for clarity):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<MaltegoMessage>
    <MaltegoTransformResponseMessage>
        <Entities>
            <Entity Type="mypackage.MypackageEntity">
                <Value>Hello Phil!</Value>
                    <Weight>1</Weight>
                    <AdditionalFields>
                        <Field DisplayName="Field 1" MatchingRule="strict" Name="mypackage.field1">2</Field>
                        <Field DisplayName="Field N" MatchingRule="strict" Name="mypackage.fieldN">test</Field>
                    </AdditionalFields>
            </Entity>
        </Entities>
    </MaltegoTransformResponseMessage>
</MaltegoMessage>

You may be wondering where those fields (mypackage.field1 and mypackage.fieldN) came from? Simple, from here:

1
2
3
4
# …
    e.field1 = 2
    e.fieldN = ‘test’
# …

Once dotransform is called, the data mining logic does it’s thing and adds entities to the response object if necessary. Finally, the response is returned and dispatcher serializes the object into XML. What about the decorators (@configure and @superuser)? Read on…

5.3 – canari install-package Magic (@configure)

So how does canari install-package figure out how to install and configure the transform in Maltego’s UI? Simple, just use the @configure decorator on your dotransform function and canari install will take care of the rest. The @configure decorator tells canari install-package how to install the transform in Maltego. It takes the following named parameters:

  • label: the name of the transform as it appears in the Maltego UI transform selection menu
  • description: a short description of the transform
  • uuids: a list of unique transform IDs, one per input type. The order of this list must match that of the inputs parameter. Make sure you account for entity type inheritance in Maltego. For example, if you choose a DNSName entity type as your input type you do not need to specify it again for MXRecord, NSRecord, etc.
  • inputs: a list of tuples where the first item is the name of the transform set the transform should be part of, and the second item is the input entity type.
  • debug: Whether or not the debugging window should appear in Maltego’s UI when running the transform.

Let’s take a look at the code again from the example above:

1
2
3
4
5
6
7
8
9
10
# …
@configure(
    label=‘To MypackageEntity [Hello World]‘,
    description=‘Returns a MyPackageEntity entity with the phrase "Hello Word!"’,
    uuids=[ ‘mypackage.v2.MyPackageEntityToPhrase_HelloWorld’ ],
    inputs=[ ( ‘Mypackage’, Person ) ],
    debug=True
)
def dotransform(request, response):
# …

The example above tells canari install-package to process the transform in the following manner:

  1. The name of the transform in the transform selection context menu should appear as To MypackageEntity [Hello World] in Maltego’s UI.
  2. The short description of the transform as it appears in Maltego’s UI is Returns a MyPackageEntity entity with the phrase "Hello Word!".
  3. The transform ID of the transform in Maltego’s UI will be mypackage.v2.MyPackageEntityToPhrase_HelloWorld. and will only work with an input entity type of Person belonging to the Mypackage transform set.
  4. Finally, Maltego should pop a debug window on transform execution.

What if we wanted this transform to work for entity types of Location, as well. Simple, just add another uuid and input tuple like so:

1
2
3
4
5
6
7
8
9
10
# …
@configure(
    label=‘To MypackageEntity [Hello World]‘,
    description=‘Returns a MyPackageEntity entity with the phrase "Hello Word!"’,
    uuids=[ ‘mypackage.v2.MyPackageEntityToPhrase_HelloWorld’, ‘mypackage.v2.MyPackageEntityToLocation_HelloWorld’ ],
    inputs=[ ( ‘Mypackage’, Person ), ( ‘Mypackage’, Location ) ],
    debug=True
)
def dotransform(request, response):
# …

Now you have one transform configured to run on two different input entity types (Person and Location) with just a few lines of code and you can do this as many times as you like! Awesome!

5.4 – Running as Root (@superuser)

At some point you may want to run your transform using a super-user account in UNIX-based environments. Maybe to run something cool like Metasploit or Nmap. You can do that simply by decorating dotransform with @superuser:

1
2
3
4
5
6
7
# …
@superuser
@configure(
# …
)
def dotransform(request, response):
# …

This will instruct dispatcher to run the transform using sudo. If dispatcher is not running as root a sudo password dialog box will appear asking the user to enter their password. If successful, the transform will run as root, just like that!

5.5 – Renaming Transforms with canari rename-transform

Alright, so you got a bit excited and decided to repurpose the helloworld transform module to do something cool. In you’re bliss you decided to change the name of the transform module to mycooltransform.py. So you’re all set to go, right? Wrong, you’ll need to change the entry in the __all__ variable (i.e. 'helloworld' -> 'mycooltransform') in src/mypackage/transforms/__init__.py, first. Why? Because canari install-package will only detect transforms if they are listed in the __all__ variable of the transform package’s __init__.py script. You can do this quite simply by running:

1
2
3
4
5
6
$ pwd
/home/user1/
$ canari rename-transform helloworld mycooltransform
renaming transform ‘helloworld’ to ‘mycooltransform’
updating __init__.py
done!

5.6 – Creating More Transforms with canari create-transform

So you want to create another transform but you want to be speedy like Gonzalez. You don’t want to keep writing out the same thing for each transform. No problem, canari create-transform will give you a head start. canari create- transform generates a bare bones transform module that you can hack up to do whatever you like. Just run canari create-transform in the src/mypackage/transforms directory, like so:

1
2
3
4
5
$ cd src/mypackage/transforms
$ canari create-transform mysecondcooltransform
creating file ./mysecondcooltransform.py…
updating __init__.py
done!

No need to add the entry in __init__.py anymore because canari create-transform does it for you automagically.

5.7 – Transform Development for the Paterva TDS

So now that you’ve become a rockstar and people are begging you to distribute those cool transforms you developed and presented at DEF CON, you’re probably wondering if there is a way to make them more accessible to the general public. Great news! There is! By using the same technology Paterva uses to distribute the built-in transforms in Maltego, you can distribute yours too. How? Simple, first you’ll need a TDS. If you don’t have the cash for your own TDS, don’t sweat it, you can always use Paterva’s free TDS which is accessible at http://cetas.paterva.com/TDS/.

Second, you’ll need a server that is going to host your transforms (preferably a Linux-based server). Make sure that the server is remotely accessible by the TDS because it will be queried by the TDS itself. Once you have your TDS and server set up, setting up your Maltego transform kingdom is right around the corner.

5.7.1 – Making Your Transforms Remote

Before we can go ahead and make our transforms remote, you’ll want to make a few simple changes:

  1. Wherever there is an import config statement, you’ll need to get rid of it.
  2. Add the config as the third parameter in the dotransform() signature. This is known as the V2 signature specification in Canari.
  3. You’ll need to add remote=True to the @configure decorator for each transform that you wish to make remote.
  4. You’ll need to get rid of any imports that may involve rendering a GUI component (e.g. message box, window, etc.).

Let’s take our previous example and covert it to the V2 specification and make it remote:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/usr/bin/env python

from canari.maltego.entities import Person
from canari.maltego.utils import debug, progress
from canari.framework import configure #, superuser
from common.entities import MypackageEntity

# …
#@superuser
@configure(
    label=‘To MypackageEntity [Hello World]‘,
    description=‘Returns a MyPackageEntity entity with the phrase "Hello Word!"’,
    uuids=[ ‘mypackage.v2.MyPackageEntityToPhrase_HelloWorld’ ],
    inputs=[ ( ‘MyPackageEntity’, Person ) ],
    remote=True
)
def dotransform(request, response, config):
    # Report transform progress
    progress(50)
    # Send a debugging message to the Maltego UI console
    debug(‘This was pointless!’)

    # Create MyPackageEntity entity with value set to ‘Hello <request.value>!’
    e = MypackageEntity(‘Hello %s!’ % request.value)

    # Setting field values on the entity
    e.field1 = 2
    e.fieldN = ‘test’

    # Update progress
    progress(100)

    # Add entity to response object
    response += e

    # Return response for visualization
    return response

That’s it! Now you’re ready to deploy this transform to your transform application server.

5.7.2 – Deploying Your Remote Transforms

You have two options to run your transforms on your transform application server. The first is to use the canari run-server. This is more for testing than production purposes. The second is to use the robust WSGI application, Plume.

To start, you’ll want to install your transform package on the remote server that will be hosting the transforms. This is done by running the setup.py script. Once installed, copy the necessary configuration files (canari.conf, mypackage.conf, etc.) to the working directory where you will be running the server from:

1
2
$ cp canari/src/canari/resources/etc/canari.conf .
$ cp mypackage/src/canari/resources/etc/mypackage.conf .

In canari.conf you’ll want to modify the following line:

1
configs=mypackage

Alternatively, you could have copied the configuration files from your ~/.canari directory on your local machine to your remote server.

To run the transform application server using canari run-server, all you need to do is run it using canari run-server --disable-ssl mypackage:

1
2
3
4
5
$ canari run-server –disable-ssl mypackage
Loading transform packages…
Loading transform package mypackage.transforms
Loading mypackage.transforms.helloworld at /mypackage.transforms.helloworld…
Starting web server on :80

To run the transform application server using Plume, you’ll need a WSGI container like mod_wsgi for Apache, or Twisted. Simply copy the plume.wsgi and plume.py files from the canari/src/canari/resources/tds directory to your WSGI container’s path as well as all the necessary configuration files. In the canari.conf file, add the following entry:

1
modules=mypackage

Once configured, run the application as instructed for the WSGI container of your choosing. For example, for Twisted, all you need to do is twistd web --wsgi plume.app. For Apache, you’ll need to add the following lines to your configuration file, assuming your document root is at /var/www and that you’ve compiled and installed mod_wsgi:

1
2
LoadModule wsgi_module libexec/apache2/mod_wsgi.so
WSGIScriptAlias / /var/www/plume.wsgi

The last step in this process is to configure the TDS. To do this, you’ll want to take a look at pages 9-11 in the Paterva TRX documentation found here: https://www.paterva.com/web6/documentation/TRX_documentation20130403.pdf.

5.7.3 – “config” and the TDS

You may be wondering what needs to be done to all the config references in your transform code. The good news is nothing! While configuring your transforms in the TDS, make sure to replace / with . for any transform settings you wish the client to control. If these settings are not provided by the client, then they will be provided by the local transform configuration file. If you absolutely require the client to specify certain settings, you will need to program that into your transform.

Navigate:Next