Send SVG to server

Using javascript, I ended up just parsing my svg html as a string and sending it with my post data.

    var svg = document.getElementsByTagName('svg')[0];
    var xml = (new XMLSerializer()).serializeToString(svg);

Include it in your post data and send it to the server…

Abstract Security Filter on Controllers

What i wanted to achieve was a generic way of determining if a user in my system was suppose to have access to a certain part of the system. I’m sure there are some really nice plugins or stuff that achieves this, but the way i achieved this functionality was to implement a controller based access filter. This way i can limit all actions in that controller simply by defining action: ‘*’

def filters = {
    all(controller: '*', action: '*') {
        before = {
            def privileged = ......//in what ever way we get ahold of this value (e.g database or session)

            def controllerArtefact = grailsApplication.getArtefactByLogicalPropertyName("Controller", "$controllerName")
            def controllerClass = controllerArtefact.getClazz()
            try {
                if(controllerClass?.ENTITY_CLASS.equals("LIMITED-ACCESS")) {
                    if(!privileged) {
                        redirect(controller: "unauthorized", action:"unauthorized")
                    }
                }
            }
            catch(MissingPropertyException e) {
                //something appropriate
            }
        }
    }
}

The only thing left to do is to add the property ENTITY_CLASS (or what ever you want to name it) to your controller which you want to limit access to.

static final ENTITY_CLASS = ‘LIMITED-ACCESS’

Username regex spring-security-plugin

Working with spring security plugin in grails, i wanted to narrow down the username possibilities to a “simple name” or an email. This was easily achieved by defining a constraint in my User class provided by the spring security plugin.

The first part of the regex evaluates the email, (not 100% sure but i think against RFC 2822), and the second parts after the OR | will evaluate the “simple name”.

(?:[a-z0-9!#$%&’*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&’*+/=?^_`{|}~-]+)*|”(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*”)@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])|(?:^[a-zA-Z0-9_\s\.]+$)

 

In my User.groovy class my final constraint was defined as follows:

static constraints = {
		username blank: false, unique: true, size: 2..32, matches: '(?:[a-z0-9!#$%&\'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&\'*+/=?^_`{|}~-]+)*|"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])|(?:^[a-zA-Z0-9_\\s\\.]+$)'
		password blank: false, size: 6..64
		email blank: false, unique:true, email:true
		firstName nullable:true, blank:true
		lastName nullable:true, blank:true
	}

Grails Render Pretty JSON

Finding this information saved me a lot of time http://www.intelligrape.com/blog/2012/07/16/rendering-json-with-formatting/

In grails we we often like to render our grails output like so:

render json as JSON

Grails converter will render the json without any formatting, which sometimes (e.glarge json files) can be hard to read.

To render our json in a pretty way we can do the following:

def json = doc as JSON
json.prettyPrint = true
json.render response

the prettyPrint option returning the grails converter json will also apply for e.g when returning it as an outputstream back to the user.

def json = doc as JSON
json.prettyPrint = true
response.setHeader("Content-disposition","attachment;filename=\"" + fileName + "\"");
response.setContentType("application/json");
response.outputStream << json

Grails truncate taglib

Ran into this taglib created by Adam Creeger which lets you truncate strings in your grails views.
It was exactly what i was looking for except for the missing possibility of also clicking it to extend and show the full text. Here is my updated version for achieving this.

class TruncateTagLib {
private static final String ELLIPSIS = '...'
	def truncate = { attrs, body ->
		def maxLength = attrs.maxlength
		if (maxLength == null || !maxLength.isInteger() || maxLength.toInteger() <= 0) {
			throw new Exception("The attribute 'maxlength' must an integer greater than 3. Provided value: $maxLength")
		} else {
			maxLength = maxLength.toInteger()
		}
		if (maxLength <= ELLIPSIS.size()) {
			throw new Exception("The attribute 'maxlength' must be greater than 3. Provided value: $maxLength")
		}
		if (body().length() > maxLength) {
			out << /<abbr id="$attrs.id" title="${body().encodeAsHTML()}" class="ellipsis" 
						onclick="javascript:if($('#$attrs.id').hasClass('ellipsis')) {
							$('#$attrs.id').html('${body()}'); $('#$attrs.id').removeClass('ellipsis')
						} else { 
							$('#$attrs.id').html('${body()[0..maxLength - (ELLIPSIS.size() + 1)].encodeAsHTML()}$ELLIPSIS'); 
							$('#$attrs.id').addClass('ellipsis') }">
					${body()[0..maxLength - (ELLIPSIS.size() + 1)].encodeAsHTML()}$ELLIPSIS<\/abbr>/
		} else {
			out << body().encodeAsHTML()
		}
	}
}

and in your view gsp simply use it as:

<g:truncate id="abbr_${it?.id}" maxlength="50">${it?.text}</g:truncate>

where it?.id is a unique identifier

Grails exception in gsp

I just came across a neat way of handling potential exceptions for when using taglibs or similar in your gsp code.

In my problem scenario i had defined some generic (twitter-bootstrap) modals that were to be pre-rendered with some form data based on a template located in each controller view.
The only problem was that this template was not used in each and every controller view, resulting in that navigating in some controllers raised an exception due to missing templates.

The way i approached this was to wrap the portion that does the rendering of the template form in a try-catch block, hence catching any exceptions raised due to missing the form.

<div id="add" class="modal hide fade">
    <div class="modal-header">
        <button type="button" class="close" data-dismiss="modal" aria-hidden="true">&times;</button>
        <h3>Add</h3>
    </div>
    <div class="modal-body">
        <%
            try{ %>
                ${render(template:"form") }
            <%} catch(Exception e){%>
            <%}
        %>
    </div>
    <div class="modal-footer">
        <button data-dismiss="modal" aria-hidden="true">Close</button>
        <button id="btn-save">Save changes</button>
    </div>
</div>

Grails asynchronous view loading with remoteFunction

This is in no way anything new, i just wanted to share an example of a really nice approach (imo) on how you can load some operation heavy rendering asynchronously using the grails remoteFunction.
Assuiming that you have a database with a large dataset that you want to aggregate and render in some fashion on a gsp page , but you do not want to stall the pageload opon navigation.

In our gsp we could add something similiar to this..

<div>
    <span class="data-loading-spinner"><r:img uri="/images/spinner.gif"/></span>
    <div id="table_data">
    </div>
</div>

Opon pageload we will se a loading spinner waiting to be replaced with the actuall data…
Since we want to load the table data when the page has fully loaded, we do this by issuing a remote call opon document.ready

<r:script>
$(document).ready(function() {
    ${remoteFunction(
        controller:'home',
        action:'render_table_data',
	update:'table_data',
	onLoading:'$(".data-loading-spinner").removeClass("hidden");',
	onComplete:'$(".data-loading-spinner").addClass("hidden");'
    )};
});
</r:script>

The remoteFunction will show the data-loading-spinner during the issuing call, and add hidden to it as soon as it returns onComplete.

So what happens in the controller?

def render_table_data() {
    //here we do whatever to gather our data.. whether it is calling a service doing some queries to the database...

    //when finished, create a model object and render it using a separate template.
    def model = [table_data:table_data]
    render template:"table_data_template", model:model
}

table_data_template here is a separate gsp file that is in the same context as our home view.

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Data</th>
        </tr>
    </thead>
    <tbody>
        <g:each in="${table_data}">
            <tr>
                <td>${it.name}</td>
                <td>${it.calculatedData}</td>
            </tr>
        </g:each>
    </tbody>
</table>

Grails and JBoss HornetQ

I would just like to share a setup i did a while back connecting my Grails application(app.grails.version=2.2.3) sending JMS messages to a JBoss HornetQ.

There were some information but not a lot on different approaches in setting up the connection factories, some more extensive than others…

Any way… The way i ended up doing it was to define my HornetQJMSConnectionFactory as a bean defined in resources.xml. I can then use this bean to instantiate a new JmsTemplate producer which then an be used to actually send the messages.

resources.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">

<bean id="hornetConnectionFactory" class="org.hornetq.jms.client.HornetQJMSConnectionFactory">
    <constructor-arg name="ha" value="false" />
    <constructor-arg>
        <bean class="org.hornetq.api.core.TransportConfiguration">
            <constructor-arg value="org.hornetq.core.remoting.impl.netty.NettyConnectorFactory" />
            <constructor-arg>
                <map key-type="java.lang.String" value-type="java.lang.Object">
                    <!-- HornetQ standalone instance details -->
                    <entry key="host" value="localhost"></entry>
                    <entry key="port" value="5445"></entry>
                </map>
            </constructor-arg>
        </bean>
    </constructor-arg>
</bean>

<!-- ConnectionFactory Definition -->
<bean id="connectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
    <constructor-arg ref="hornetConnectionFactory" />
</bean>

<bean id="hornetqJmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
    <property name="connectionFactory" ref="connectionFactory" />
    <property name="transacted" value="false" />
    <property name="requestTimeout" value="60000" />
</bean>

<bean id="hornetq" class="org.apache.camel.component.jms.JmsComponent">
    <property name="configuration" ref="hornetqJmsConfig" />
</bean>
</beans>

Now when we have our spring bean in our context we can then from a e.g controller och service do

class SendMessageService {
    def hornetConnectionFactory
    
    def sendMessage() {
        JmsTemplate producer = new JmsTemplate(hornetConnectionFactory);
        producer.send(new HornetQQueue("myqueue"), new MessageCreator() {
            @Override
            public Message createMessage(Session session) throws JMSException {
                // Create a byte messages
                BytesMessage message = session.createBytesMessage()
                message.writeBytes("hello world".getBytes())
                return message
            }
        }
    }
}

MQTT and Android

A while back a friend and started developing an online multiplayer Geo-Racing game where a group of people are able to compete against each other in a race from Point A to Point B anywhere in the world (almost). In all its simplicity it is basically like any other running gps tracker out there, with an additional built in competitive system. This post is simply just a small introduction and overview of some of the technologies that is included in the project and a brief introduction of the backend mechanics of the game.

My project partner has written a nice post with some more detail on the technology layers and some more details on the beneficial common layering :

http://www.squeed.com/blog/2013/12/mqtt-powered-event-bus-for-android-applications/

http://www.squeed.com/blog/2013/12/common-jpa-entities-for-android-and-backend/

Why MQTT?
Because we thought it was a fun idea exploring the possibilities of creating a fully message driven system on Android. Also with its messaging being tiny, power efficient, low cost keep alive and low latency push capabilities, it is a perfect fit for the system we are building.

I will not touch upon the subject of performance, there is already an excellent post from Stephen Nicholas taking up some of these aspects http://stephendnicholas.com/archives/219.

Technology choices

There are more and more brokers out there supporting the MQTT protocol. Some of the brokers we initially tried out was ActiveMQ, ActiveMQ Apollo and Mosquitto. The natural choice fell on Mosquitto given it is a native MQTT broker with good documentation, a hot mailing list and fast turnarounds for bugs and support (http://mosquitto.org/).

I’v come across two settled opensource MQTT client libraries that can be used for the server and client setup.

  • Eclipse Paho (http://www.eclipse.org/paho/) “is the primary home of the reference MQTT clients that started at IBM”, has good documentation and supports a variety of languages
  • Fusesource MQTT – Client (https://github.com/fusesource/mqtt-client) (the one used in our project) in all its simplicity has some nice predefined ways of setting up your connecting, such as, BlockingConnection – FutureConnection and CallbackConnection

The topology

Architecture

So the architecture of this whole setup is pretty straight forward. Basically what we have from start-up is a Server application connecting and subscribing to an (what we call it) Action Topic (AT) on the Mosquitto broker. Any android application (A) connecting to the broker will start subscribing to a unique topic for that user. If we have 100 users, we will have 101 topics in the system, that may or may not actively subscribe to their own topic. This means that any action that results in a message delivery to one to many topics will result in a very fast push of information to all the consuming devices.

The message handling

Has been abstracted as a common library included by both the server and the android applications. Since the MQTT message payload is of type byte, we made the decision to define any message sent between the client and the server (and vice versa) to be consisting of a byte[] serialized java class. This consists of a top level abstract class “Application Message” which any new message type would then extend. This means that any payload sent between the client and server will be a deserializable byte payload known to both the client and server message handler.

So sending a message from the client to the server could go like this:

  1. Create a new message and add all necessary information
  2. Serialize the message to a byte[]
  3. Add the data to the buffer payload and publish mqtt message on destination topic.

Now the server will retrieve this buffer payload, deserialize it, and by Visitor Pattern determine what kind of “Application Message” where sent and operate on the message accordingly.
The beauty of this now is that the Android application will react in the exact same way. The three steps would be taken by the server for any message that might be sent to the client, and by convention the client subscribing to that topic would determine what message were sent by deserializing and dropping it through the message visitor and act on it accordingly. And for any additional processing necessary on either the server or android application we can simply override the basic behavior for the visit method for that message type and define our own logic.

The database back-end has also been included in the common packaging for both the server and the client. OrmLight (http://ormlite.com/) is a relatively lightweight ORM java package for persisting java objects in an SQL database. Here we define all our database entities by JPA annotations and define our DAO layer. The beauty of this is that we are simply able to make use of the same dao for both the server and android back-end. Since the servers main objective is “only” to mirror and persist the state of our complete network the way we define our CRUD can be commonly shared.

The servers main objective is to persist and store information. In some scenarios it also is responsible for forwarding messages between users when they lack information to send the message directly to the user themselves. It is also responsible for Registration and Login. Since MQTT by nature is asynchronous, we have implemented a custom synchronous messaging between the client and server using callbacks for handling logins and registration.

The android client will periodically push its information to one or more topics when active. The messages sent by the android app will be published to either client, client&server or server. When publishing the information, MQTT supports three different types of Quality Of Service (QoS). When choosing in what QoS the message should be sent, there will be an affect on the applications battery consumption etc since there are more or less messages being sent in the background.

Information below on QoS is taken from the paho documentation: http://www.eclipse.org/paho/files/mqttdoc/Cclient/qos.html

  • QoS0, At most once: The message is delivered at most once, or it may not be delivered at all. Its delivery across the network is not acknowledged. The message is not stored. The message could be lost if the client is disconnected, or if the server fails. QoS0 is the fastest mode of transfer. It is sometimes called “fire and forget”. The MQTT protocol does not require servers to forward publications at QoS0 to a client. If the client is disconnected at the time the server receives the publication, the publication might be discarded, depending on the server implementation.
  • QoS1, At least once: The message is always delivered at least once. It might be delivered multiple times if there is a failure before an acknowledgment is received by the sender. The message must be stored locally at the sender, until the sender receives confirmation that the message has been published by the receiver. The message is stored in case the message must be sent again.
  • QoS2, Exactly once: The message is always delivered exactly once. The message must be stored locally at the sender, until the sender receives confirmation that the message has been published by the receiver. The message is stored in case the message must be sent again. QoS2 is the safest, but slowest mode of transfer. A more sophisticated handshaking and acknowledgement sequence is used than for QoS1 to ensure no duplication of messages occurs.

Message aggregation

In all its simplicity we want to assure that any message sent will reach its destination. How we propagate the messages in order for all involved parties to receive their messages is a challenge. The following is an example of how the message propagation works when we do a race-invite to multiple users.

race invite1

A1 sends a race invite to A2 and A3 containing the race information and a list of anyone that might need further information regarding this race (information so that A2 and A3 may communicate since they might not be friends)

race invite2

A2 Accepts the invitation by creating the race in its local database, return his information to A1, A3 and the server so that the two get all information and the server can persist.

race invite3

Finally A3 also accepts the race invite, creating the race in his local database and sending his information to both A1, A2 and the server for persistence.

Whats left out in these examples is the actual retry policy that is underlying for both the server and the android client. There is no support in MQTT (AFAIK) that takes care of persisting and automatically retrying to publish any failed publications. Lets say that the mosquitto broker were to be offline at the point of a message publication, the retry to send this message needs to be handled on client level, hence we have written our own retry policy to take care of these events.

To sum up…

So far having chosen MQTT as our primary communications protocol it has been  a fast and fun experience learning some new aspects diverging from the more common ways of communicating in a distributed environment. The project is still in working progress, so we are still eager for any interesting problems that might arise.

Mule and FTP, downloading multiple files from within a Callable component

Versions:

  • MuleESB v.3.2.1

This is just a solution i wrote in order to dynamically download multiple files from an FTP server programmatically from within a Mule Callable Component.

This is most likely not the most kosher way of getting multiple files from an FTP, however it did give me the satisfactory results since i was able to selectively choose which files to download and one by one send out in the flow construct.

The tricky part was that i wanted to get files with some specific prefix.. e.g  we are only interested in the files starting with the letter ‘A’. So for a file list with 3 files  A-X.xml, B-Y.json, A-XX.xml, we only want to download file A-X and A-XX.

The first part is implementing the Callable interface which provides us with the onCall(MuleEventContext eventContext) method. In this example we are providing the prefix in the payload fetching it from the muleMessage in the eventContext. We then get a hold of the filelist on the ftp by using a FTPClient lib and issuing listNames.

For any file matching our prefix pattern, we get the file with the getFile method and send it out on the flow construct to our destination flow, in this case to a vm endpoint name processing.

public class GetFiles implements Callable {
	public Object onCall(MuleEventContext eventContext) throws Exception {
		// Get the filename
		Object filenameprefix = (Object) eventContext.transformMessage(Object.class);
		MuleEventContext eventCtx = RequestContext.getEventContext();

		MuleMessage msg = null;
                //iterate through the filelist on the ftp server.
		for(String filename : getFileList()) {
			if(filename.matches(filenameprefix+".*")) {  //if we find a file with a filename matching our regex
				msg = getFile(eventCtx, filename);  //get the file from the ftp
				msg.setOutboundProperty(FileConnector.PROPERTY_ORIGINAL_FILENAME, filename);
				msg.setOutboundProperty(FileConnector.PROPERTY_FILENAME, filename);

				if(msg.getPayloadAsString() != null || msg.getPayloadAsString() != "") {
					eventCtx.sendEvent(msg, "vm://processing");  //send the message to the destination endpoint
				}
				else {
					throw new DefaultMuleException("Payload corrupt");
				}
			}
		}
		return msg;
	}

	public MuleMessage getFile(MuleEventContext eventCtx, String filename) throws MuleException {
		//build an ftp endpoint
		EndpointBuilder ftpEndpointBuilder = RequestContext.getEvent().getMuleContext().getRegistry().lookupEndpointFactory()
				.getEndpointBuilder("ftp://" + getFtp_user() + ":" + getFtp_password() + "@" + getFtp_location()
						+ 	":" + getFtp_port() + "/" + getFtp_path());

		//get a specified file based on the filename. Add a message processor to the ftpEndpointBuilder with a FileNameWildcardFilter.
		ftpEndpointBuilder.addMessageProcessor(new org.mule.routing.MessageFilter(
				new org.mule.transport.file.filters.FilenameWildcardFilter((filename))));

		MuleMessage msg = eventCtx.requestEvent(ftpEndpointBuilder.buildInboundEndpoint(), timeout); //fetch the file

		//if we could not find the file, we throw back an error
		if (msg == null) {
			throw new DefaultMuleException("File -- " + filename + " -- not found");
		}
		return msg;
	}

	/*
	 * Get a list of all files available at the specified path
	 */
	public String[] getFileList() {
		FTPClient client = new FTPClient();

		String[] names = null;
		try {
			client.connect(getFtp_location(), Integer.parseInt(getFtp_port()));
			client.login(getFtp_user(), getFtp_password());
			names = client.listNames(getFtp_path()); //list the names of all files on the ftp.
			client.logout();
		} catch (IOException e) {
			e.printStackTrace();
		} finally {
			try {
				client.disconnect();
			} catch (IOException e) {
				e.printStackTrace();
			}
		}
		return names; //return the list of names
	}
}