Sonatype Nexus not returning the latest releases through REST interface

Sonatype's Nexus Maven repository behaves strangely after using Rebuild Metadata feature from the Nexus UI. For some reason this feature updates all maven-metadata.xml files under repository and adds this line to all of them

<latest>0.0.48</latest>

The added line breaks Nexus's REST query API so that it always returns the version given in the latest element instead of the real latest released version. So, for example this query would always return version 0.0.48, if the latest element is in maven-metadata.xml file.

$ curl
"http://<hostname>:<port>/nexus/service/local/artifact/maven/redirect?r=releases-repository-name&g=com.group&a=artifact-name&e=jar.md5&v=LATEST

To fix this, you can run the following command under the broken repository to remove all latest elements.

$ find . -name maven-metadata.xml -exec sed -i -r "/<latest>.+<\/latest>/d" \{\} \;

An alternative way to query the latest releases is the following.

$ curl
"http://<hostname>:<port>/nexus/service/local/artifact/maven/redirect?r=releases-repository-name&g=com.group&a=artifact-name&e=jar.md5&v=RELEASE

This query works even with the latest element.

It seems that latest element is not affecting queries of the latest SNAPSHOTs.

For more information about this subject.

• https://oss.sonatype.org/nexus-restlet1x-plugin/default/docs/path__artifact_maven_redirect.html
• http://articles.javatalks.ru/articles/3

How to make Ubuntu play music over bluetooth

After installing latest Ubuntu version 14.04 (Trusty Tahr) I found out by accident that Ubuntu is able to play music from bluetooth source. Here's how it works on my laptop (Sony Vaio VPCEA2S1E, perhaps there's differences in the bluetooth adapters?).

Check has PulseAudio already loaded module-bluetooth-discover.
$ pactl list | grep -i module-bluetooth-discover

If this command does not return anything, you should load the module with the following command.
$ pactl load-module module-bluetooth-discover

Now the first command shows that the module is loaded.
$ pactl list | grep -i module-bluetooth-discover
Name: module-bluetooth-discover

After executing the command, you should be able to add your for example your phone as the source of audio. In my Windows Phone Lumia 1020, I navigate to Settings -> Accessories -> add using bluetooth. If you have music playing on your phone and you connect to your Ubuntu via bluetooth, your phone will show text connected music. And the music is coming out of your computer's speakers!

I also discovered that in the latest Ubuntu version my bluetooth keyboard Logitech diNovo Edge settings have to be adjusted manually. The mouse pointer does not move fast enough when using keyboard's touchpad. In the previous version Ubuntu 12.04 I was able to set the speed and acceleration of pointer fast enough from System Settings UI. In 14.04, I cannot. However, it's easy to change the acceleration and speed of pointer from command line. Here's the command.
$ xset m 3 1

All in all, I'm very happy with the latest Ubuntu release!

Setting up Logback with nice default appender

I spent some time looking for a solution to make Logback log everything to console in development environment and still have easy setup for logging into file in production environment.

So, my goal was that if I execute my application without any extra Logback configuration, everything goes to console. And in case configuration file exists in the environment, Logback would use that and forget the default configuration.

The default console logging configuration is in the root of classpath and is named as logback.xml (the file Logback looks for by default). Here's the contents the file.

<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="30 seconds">

  <define name="INCLUDED_FILE_EXISTS" class="ch.qos.logback.core.property.FileExistsPropertyDefiner">
    <path>${configpath}/included-logback.xml</path>
  </define>
  
  <if condition='property("INCLUDED_FILE_EXISTS").equals("true")'>
    <then>
      <!-- this configuration is used in other than development environments i.e. custom config per environment -->
      <include file="${configpath}/included-logback.xml"/>
    </then>
    <else>
      <!-- This configuration is used only in development enviroment -->
      <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
          <pattern>%d{HH:mm:ss.SSS} %-5level %logger{35} - %msg%n</pattern>
        </encoder>
      </appender>
      <root level="WARN">
        <appender-ref ref="STDOUT" />
      </root>
      <logger name="fi.foo.bar.package" level="DEBUG"/>
    </else>
  </if> 
</configuration>

This configuration has two important sections. First one is the use of FileExistsPropertyDefiner. It checks the existence of the given file in file system and sets value of property INCLUDED_FILE_EXISTS to true or false. If you want, you can also check existence of a file on classpath by using ResourceExistsPropertyDefiner. The variable configpath is defined in system properties.  Second interesting section is where the INCLUDED_FILE_EXISTS property is used to include the external file. If the file was found, its configuration is used and default configuration is discarded completely.

Here is example of the included file. The file has name included-logback.xml.

<included scan="true">
  <!-- This file is scanned so you can update it while application is running -->
  <appender name="FILELOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${configpath}/logs/application.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>${configpath}/logs/application-%d{yyyy-MM-dd}.log.gz</fileNamePattern>
    </rollingPolicy>
    <encoder>
      <pattern>%d{HH:mm:ss.SSS} %-5level %logger{35} - %msg%n</pattern>
    </encoder>
  </appender>

  <root level="INFO">
    <appender-ref ref="FILELOG" />
  </root>
</included>

Creating a bootable Xubuntu USB stick and installing Xubuntu on another USB stick

A friend of mine was complaining that his Asus Eee 1015BX is running too slow with Windows 7 Starter that came pre-installed with the laptop. It's no wonder that Windows is slow, because the laptop has only 1GB of memory. As a great (Ubuntu) Linux evangelist, I wanted to help the poor guy and give him a chance to test Ubuntu without any risks.

Because Ubuntu would probably be too heavy for such low end hardware, I thought that Xubuntu would serve better. I didn't want to change anything on the laptop. I just wanted to offer the Ubuntu experience without any risks. I decided that the best way to achieve this would be to install Xubuntu on USB disk.

To install Xubuntu on USB disk, you need installation media. Well, this laptop has only two USB ports (of course also LAN), so I cannot use DVD as in my previous installations. First problem was to create Xubuntu installation media. For some reason Ubuntu Startup Disk Creator doesn't allow user to create bootable USB stick from Xubuntu ISO image. Luckily there's an alternative, UNetbootin, which worked perfectly.

Next step was to run the installation software. You cannot set Eee's BIOS to boot from USB, but you can hit escape key during start-up to change booting options. After I got Eee to boot from USB, I inserted another USB stick, which would serve as the installation target media.

I'm not going through the details step-by-step, because installer is quite easy to use. I created /, /boot and /home partitions and no swap. It's important to notice that you shouldn't use USB stick to store a swap partition. Firstly, swap on USB is slow and secondly, it will - according to internet discussions - destroy your USB stick after some time. Installation took a lot of time, almost an hour. It would never take that long on a hard drive.

After the installation, there's some things you can do to optimise the speed of the system and the lifetime of the USB drive. Here's what I did.

Edit /etc/fstab. Add noatime flag to / and /home to avoid updating access times on files. This should speed things up  and help to avoid extra wearing of the USB stick. Here's an example.
# / was on /dev/sdb5 during installation
UUID=ae7eb15f-a65b-4e2b-a008-20a2b41ce72a /               ext4    noatime,errors=remount-ro 0       1
# /boot was on /dev/sdb1 during installation
UUID=190d4960-d773-451f-bbb3-b1a8bd39238b /boot           ext4    defaults        0       2
# /home was on /dev/sdb6 during installation
UUID=2fb3fc52-6155-4fdf-89f5-d991bca484fc /home           ext4    defaults,noatime        0       2

Disable mlocate so that it won't re-index the whole disk every day.
sudo chmod -x /etc/cron.daily/mlocate

Disable journaling from / and /home (you need to do these in single user mode or by starting Linux from some other media like the USB stick containing installation media, because the partitions cannot be mounted during the operation). By disabling journaling, you may end up with corrupted file system if the system crashes. On the other hand, you reduce the number of writes on the USB drive which should make things faster and prolong the lifetime of USB drive.
sudo tune2fs -O ^has_journal /dev/sdb5
sudo tune2fs -O ^has_journal /dev/sdb6

By the way, you can also prepare the drives already beforehand and format the partitions for ext4 without journaling with the following command. I did this after my first installation attempt, because I hoped it would speed up the slow installation process (I'm not sure did it help).
sudo mke2fs -t ext4 -O ^has_journal /dev/sdb6

After editing the file system parameters, it's a good idea to check that the file system is ok (-f for forcing the check)
sudo e2fsck -f /dev/sdb5
sudo e2fsck -f /dev/sdb6


Here's what I learned from all of this. Xubuntu is much much faster than Windows 7 on the Asus Eee. That was no surprise. All the hardware worked properly. After installing the proprietary AMD display drivers, battery consumption went down and battery life is at least as good as on Windows. Although it takes some time to install Xubuntu from USB to USB, it's a very good way to make a test drive with Xubuntu. You don't risk anything and user can always go back to Windows without any extra hassle.

Although Xubuntu worked well, my USB stick didn't. I bought a tiny Kingston 32GB Datatraveller Micro to serve as installation target media. The Datatraveller Micro is so small that you can carry it attached to the laptop all the time. Unfortunately, it's also slow making Xubuntu work very slowly during disk intensive operations. I compared the write speed of Datatraveller Micro to Datatraveller 100 on USB 2 port. Datatraveller Micro's write speed was only 6MB/s whereas Datatraveller 100's write speed was 22MB/s. Although the physical dimensions of Datatraveller Micro are perfect for running laptop Linux from it, the speed of the drive makes it questionable for this purpose.

This story may have a happy ending, because it seems that Ubuntu is getting yet another happy user in the very near future (after I install Xubuntu on Eee's hard drive)!

Easy JUnit testing with Elastic Search

It was quite difficult to find good examples on JUnit testing (this is more like integration testing than unit, but nevertheless JUnit is used) Elastic Search code. Here's my attempt to fix this issue.

The idea is to start a standalone Elastic Search instance in the test. This way we don't have to make sure that every developer has access to Elastic Search instance running somewhere out of build's control. The downside of starting an instance inside test is that it may get quite slow to run the test. However, that's another problem to tackle.

I'm using Jersey HTTP client in this example to connect to Elastic Search. Any other client works as well.

Maven POM 

Maven POM that's needed to execute my example.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>fi.elastic-search.test</groupId>
  <artifactId>elastic-junit</artifactId>
  <name>elastic-junit</name>
  <packaging>jar</packaging>
  <version>1.0.0-BUILD-SNAPSHOT</version>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>0.20.3</version>
    </dependency>
    <dependency>
      <groupId>com.sun.jersey</groupId>
      <artifactId>jersey-client</artifactId>
      <version>1.17</version>
</dependency> 
  </dependencies>
</project>

JUnit test code 

Here's the Java code starting standalone Elastic Search instance and creating an index.

import java.io.File;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;

import javax.ws.rs.core.MediaType;

import org.elasticsearch.common.io.FileSystemUtils;
import org.elasticsearch.common.settings.ImmutableSettings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.node.Node;
import static org.elasticsearch.node.NodeBuilder.nodeBuilder;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.UniformInterfaceException;
import com.sun.jersey.api.client.WebResource;
import com.sun.jersey.api.client.config.ClientConfig;
import com.sun.jersey.api.client.config.DefaultClientConfig;
import com.sun.jersey.api.client.filter.LoggingFilter;


public class ElasticSearchTest {
    private Client client;
    private static String testIndexName = "junitindexname";
    private static final String HTTP_BASE_URL = "http://localhost";
    private static final String HTTP_PORT = "9205";
    private static final String HTTP_TRANSPORT_PORT = "9305";
    private static final String elasticSearchBaseUrl = HTTP_BASE_URL + ":" + HTTP_PORT;

    private static Node node;

    @BeforeClass
    public static void startElasticSearch() throws Exception {
        final String nodeName = "junittestnode";

        Map settingsMap = new HashMap();
        // create all data directories under Maven build directory
        settingsMap.put("path.conf", "target");
        settingsMap.put("path.data", "target");
        settingsMap.put("path.work", "target");
        settingsMap.put("path.logs", "target");
        // set ports used by Elastic Search to something different than default
        settingsMap.put("http.port", HTTP_PORT);
        settingsMap.put("transport.tcp.port", HTTP_TRANSPORT_PORT);
        settingsMap.put("index.number_of_shards", "1");
        settingsMap.put("index.number_of_replicas", "0");
        // disable clustering
        settingsMap.put("discovery.zen.ping.multicast.enabled", "false");
        // disable automatic index creation
        settingsMap.put("action.auto_create_index", "false");
        // disable automatic type creation
        settingsMap.put("index.mapper.dynamic", "false");

        removeOldDataDir("target/" + nodeName);

        Settings settings = ImmutableSettings.settingsBuilder()
                .put(settingsMap).build();
        node = nodeBuilder().settings(settings).clusterName(nodeName)
                .client(false).node();
        node.start();
    }

    private static void removeOldDataDir(String datadir) throws Exception {
        File dataDir = new File(datadir);
        if (dataDir.exists()) {
            FileSystemUtils.deleteRecursively(dataDir, true);
        }
    }

    @AfterClass
    public static void stopElasticSearch() {
        node.close();
    }

    @Before
    public void initialize() {
        // create client for each test
        ClientConfig clientConfig = new DefaultClientConfig();
        client = Client.create(clientConfig);
        client.addFilter(new LoggingFilter(System.out));
    }
    
    @Test
    public void testCreateIndex() {
        WebResource service = client.resource(elasticSearchBaseUrl);
        // check first that does the index already exist
        ClientResponse clientResponse = service.path(testIndexName).head();
        if (clientResponse.getClientResponseStatus().equals(ClientResponse.Status.OK))
        {
            Assert.fail("Index exists already");
        }

        String indexJson = 
                "{\"settings\" : {\"index\" : {\"number_of_shards\" : 1,\"number_of_replicas\" : 0}}}";
        try {
            String response = 
                    service.path(testIndexName).
                    queryParam("refresh", "true").
                    queryParam("timeout","5m").
                    accept(MediaType.APPLICATION_JSON).
                    put(String.class, indexJson);
            if (!response.contains("ok")) {
                Assert.fail("Creating index failed. IndexName: " + testIndexName);
            }
        } catch (UniformInterfaceException e) {
            // failed due to Client side problem
            throw e;
        }

        // wait for Elastic Search to be ready for further processing
        HashMap statusParameters =
                new HashMap();
        final String timeout = "30s";
        statusParameters.put("timeout", timeout);
        statusParameters.put("wait_for_status", "green");
        String statusResponse = status(statusParameters);
        if (statusResponse.contains("red")) {
            Assert.fail("Failed to create index");
        }
    }
    
    public String status(final Map optionalParameters) {
        WebResource service = client.resource(elasticSearchBaseUrl);
        WebResource webResource = service.path("_cluster").path("/health");
        for (Entry entry : optionalParameters.entrySet()) {
            webResource = webResource.queryParam(entry.getKey(),entry.getValue());
        }
        return webResource.get(String.class);
    }

}

Some tomcat7-maven-plugin tricks

Here's some useful tomcat7-maven-plugin tricks that work also with other Maven plugins.

How to set java endorsed dir for tomcat7-maven-plugin

Sometimes you need to use Java endorsed standards override mechanism with Maven plugins. This can be accomplished simply by setting proper MAVEN_OPTS value. For example

export MAVEN_OPTS="-Djava.endorsed.dirs=/some/directory/where/endorsed/jars/are"

How to change tomcat7-maven-plugin classpath order

Very often you need to have the runtime classpath in certain order when running Tomcat or other Maven plugins. I found out by trial and error that the used classpath JAR ordering depends on the order of declared dependencies in pom.xml (most likely this is documented somewhere). So, if you need to get some libraries before others, just move them in the beginning of pom.xml. Of course this does not help, if you have many plugins that need a different order for each plugin.

How to list all loaded classes and their respective JAR files

This is something I had been looking for years and was extremely happy when I finally got to know it. At least Sun Java JRE provides a command line switch for showing all the classes that are loaded into JVM and the source JAR file of those classes. It's not always easy to know what classes are loaded, especially if your dependencies have dependencies to other JAR files that are somehow conflicting (for example different version) with JAR dependencies defined elsewhere. Anyway, to list the classes and their exact source, do the following.

export MAVEN_OPTS="-verbose:class"

The output is something like this.

[Loaded javax.xml.bind.annotation.XmlElement from /home/perttu/software/jdks/jdk1.6.0_31/jre/lib/rt.jar]
[Loaded javax.xml.bind.annotation.XmlElement$DEFAULT from /home/perttu/software/jdks/jdk1.6.0_31/jre/lib/rt.jar]
[Loaded javax.xml.ws.WebFault from /home/perttu/software/jdks/jdk1.6.0_31/jre/lib/rt.jar]

With these tools, any problem related to Tomcat Maven plugin and classpath should be easy to solve.

Encrypting SWAP partition and taking previously encrypted home partition into use

Due to my personal Ubuntu release update process, I have to configure encryption of my boot and home partition manually every now and then. Because I do this regularly, I'm documenting my process here in my blog...

Here's my Ubuntu release update process.

1. Make Ubuntu installation program to format the partition that held the previous Ubuntu version
2. Format and configure existing boot partition as the new boot partition
3. Configure existing swap partition as the new swap partition
4. Install Ubuntu normally without dedicated home partition

After installation, my encrypted home partition is not accessible anymore, because it was not configured as part of the installation process. I haven't been able to configure either normal installer or alternative installer to take previously encrypted partitions into use. So, it has to be done after installation.

After installation has finished, install cryptsetup.

$ sudo apt-get install cryptsetup

Then configure /etc/fstab file by adding configuration for swap and home. For example.

/dev/mapper/sda7_crypt /home           ext4    defaults        0       2
/dev/mapper/sda5_crypt none            swap    sw              0       0

The actual values depend on your hard disk partitioning. Here's example from my laptop.

$ sudo fdisk -l

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7250e0b9

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048    24588287    12293120   27  Hidden NTFS WinRE
/dev/sda2        24588288    24793087      102400    7  HPFS/NTFS/exFAT
/dev/sda3   *    24793088    26746879      976896   83  Linux
/dev/sda4        26748926   976773119   475012097    5  Extended
/dev/sda5        26748928    34559999     3905536   82  Linux swap / Solaris
/dev/sda6        34562048   132216831    48827392   83  Linux
/dev/sda7       132218880   976773119   422277120   83  Linux

The next step is to configure /etc/crypttab by adding appropriate encryption setup. In my case it looks like this.

sda5_crypt /dev/sda5 /dev/urandom cipher=aes-cbc-essiv:sha256,size=256,swap
sda7_crypt UUID=5a9c38c3-2aa9-433b-9efd-c0e9357d0811 none luks

The swap partition setup is "universal" and it should work on any computer (of course the correct partition may differ from this example). The swap is encrypted with a key that is randomly generated on each system startup.

For the home partition you have to know the UUID of the partition. Here's one way to find it.

$ sudo blkid 
/dev/sda1: LABEL="Recovery" UUID="78F82CACF82C6A98" TYPE="ntfs" 
/dev/sda2: LABEL="System Reserved" UUID="62ACA8E6ACA8B5C7" TYPE="ntfs" 
/dev/sda3: UUID="7d526d52-018a-4b0c-9e26-64f1143cf0da" TYPE="ext4" 
/dev/sda5: UUID="0d0178b1-ade5-416f-8535-82455a8febd5" TYPE="swap" 
/dev/sda6: UUID="b0d7a0b7-0bb8-4cf9-978a-f7c6ebb2126f" TYPE="ext4" 
/dev/sda7: UUID="5a9c38c3-2aa9-433b-9efd-c0e9357d0811" TYPE="crypto_LUKS" 
/dev/sdb1: LABEL="siirtoNTFS" UUID="494640E516B11A6B" TYPE="ntfs" 
/dev/sdb2: LABEL="siirto" UUID="570857a6-4ab3-4d4b-99a2-88d383d3e588" TYPE="ext4" 

With this configuration, the system should ask for home partition's encryption key during system startup and everything should work as before.