End to End Encryption with Beanstalk

May 23rd, 2019 No comments

Beanstalk is often configured to terminate SSL at the load balancer then make the connection to the web server/application instances using unencrypted HTTP. That’s usually okay as the AWS network is designed to keep such traffic private, but under certain conditions, such as those requiring PCI compliance, DoD/government rules, or simply out of an abundance of caution, there’s a desire to have all traffic encrypted – including that between the Beanstalk load balancer and servers.

There are two approaches for implementing end to end encryption on Beanstalk:

  • Use a layer 4 load balancer (Network or Classic Elastic Load Balancer.
    Using this approach, the load balancer never decrypts the traffic. The downside is that advanced reporting isn’t possible and layer 7 features, such as session affinity, cannot be implemented.
  • Use a layer 7 load balancer (Application or Classic Load Balancer).
    Using this approach, traffic is decrypted at the load balancer. The load balancer would then re-encrypt traffic to the servers. Session affinity and traffic reporting are available.

The preferred solution is to use the layer 7 approach with an Application Load Balancer. This preference is due to the additional features the layer 7 offers, because Network Load Balancers are more expensive, and because AWS is deprecating Classic Load Balancers.

The simplest way to accomplish this goal is to use a self signed certificate on the servers and then use HTTPS from the load balancer to the server. Application Load Balancers do not currently perform validation of certificates which is why the self signed approach works and why there’s no advantage to using a CA issued certificate.

The following approach will work on any Beanstalk supported platform that uses nginx as the proxy server. This configuration is based on AWS’s documentation, but trimmed for only Application Load Balancers and to include the nginx configuration and self-signed certificate generation.

In your Beanstalk application archive, add these files:

.ebextensions/nginx/conf.d/https.conf

# HTTPS server

server {
    listen       443;
    server_name  localhost;
    
    ssl                  on;
    # The certificate is generated in generate-certificate.sh
    ssl_certificate      /etc/pki/tls/certs/server.crt;
    ssl_certificate_key  /etc/pki/tls/certs/server.key;
    
    ssl_session_timeout  5m;
    
    ssl_protocols  TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers   on;
    
    location / {
        proxy_pass  http://localhost:5000;
        proxy_set_header   Connection "";
        proxy_http_version 1.1;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto https;
    }
}

.ebextensions/nginx-https.conf

option_settings:
  aws:elasticbeanstalk:application:
    Application Healthcheck URL: HTTPS:443/
  aws:elasticbeanstalk:environment:process:default:
    Port: '443'
    Protocol: HTTPS
  aws:elbv2:listener:443:
    DefaultProcess: https
    ListenerEnabled: 'true'
    Protocol: HTTPS
  aws:elasticbeanstalk:environment:process:https:
    Port: '443'
    Protocol: HTTPS
packages:
  yum:
    bash: []
    openssl: []
files:
  "/tmp/generate_nginx_certificate.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      set -Eeuo pipefail # stop on all errors
      # These files are used by nginx, see nginx/conf.d/https.conf
      openssl genrsa 2048 > /etc/pki/tls/certs/server.key
      openssl req -new -x509 -nodes -sha1 -days 3650 -extensions v3_ca -key /etc/pki/tls/certs/server.key -subj "/CN=localhost" > /etc/pki/tls/certs/server.crt
commands:
  01generate_nginx_certificate:
    command: "/tmp/generate_nginx_certificate.sh"
Categories: Uncategorized Tags:

Change the Spring Session JDBC Serialization Method to Improve Performance

May 8th, 2019 No comments

Spring Session JDBC is a great way to allow an application to be stateless. By storing the session in the database, a request can be routed to any application server. This approach provides significant advantages such as automatic horizontal scaling, seamless failover, and no need for session affinity. By using JDBC, the database the application is already using provides the storage avoiding the need to setup and maintain other software, such as Memcache or Redis.

When Spring Session JDBC stores the session in the database, it has to serialize (convert from a Java object to a string) the session and also deserialize it (convert from a string back to a Java object). By default, it uses Java’s built in serialization.

There are numerous reasons not to use Java’s built in serialization (ObjectInputSteam / ObjectOutputStream). Oracle calls it a “Horrible Mistake” and plans to remove it in a future Java release. It’s also less performant and produces a larger serialized form than many alternatives.

Since Java serialization is (at least, for now) included in Java, it’s still commonly used, including by Spring Session JDBC. Switching to another serialization method can be a relatively quick and easy way to improve performance.

Any serialization can be used, including Jackson (which uses the JSON or XML format), Protocol Buffers, Avro, and more. However, all require work to define schemas for the data and additional configuration. In the interest of avoiding those efforts (which is especially important for legacy applications), a schemaless serializer (which is what Java’s built in serializer is) can be used such as FST (fast-serializer) or Kryo.

Switching the serializer used by Spring Session JDBC is done by defining a a bean named springSessionConversionService of type ConversionService. The following examples provide the code to use FST or Kryo.

Using FST with Spring Session JDBC

Add FST as dependency to the project. For example, using Maven:

<dependency>
    <groupId>de.ruedigermoeller</groupId>
    <artifactId>fst</artifactId>
    <version>2.56</version>
</dependency>

And these add these classes:

FstSessionConfig.java

import org.springframework.beans.factory.BeanClassLoaderAware;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.convert.ConversionService;
import org.springframework.core.convert.support.GenericConversionService;
import org.springframework.core.serializer.support.DeserializingConverter;
import org.springframework.core.serializer.support.SerializingConverter;

@Configuration
public class FstSessionConfig implements BeanClassLoaderAware {

	private ClassLoader classLoader;

	@Bean
	public ConversionService springSessionConversionService() {
		final FstDeserializerSerializer fstDeserializerSerializer = new FstDeserializerSerializer(classLoader);

		final GenericConversionService conversionService = new GenericConversionService();
		conversionService.addConverter(Object.class, byte[].class,
				new SerializingConverter(fstDeserializerSerializer));
		conversionService.addConverter(byte[].class, Object.class,
				new DeserializingConverter(fstDeserializerSerializer));
		return conversionService;
	}

	@Override
	public void setBeanClassLoader(final ClassLoader classLoader) {
		this.classLoader = classLoader;
	}
}

FstDeserializerSerializer.java

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.nustaq.serialization.FSTConfiguration;
import org.nustaq.serialization.FSTObjectOutput;
import org.springframework.core.NestedIOException;
import org.springframework.core.serializer.Deserializer;
import org.springframework.core.serializer.Serializer;

public class FstDeserializerSerializer implements Serializer<Object>, Deserializer<Object> {

	private final FSTConfiguration fstConfiguration;
	
	public FstDeserializerSerializer(final ClassLoader classLoader) {
		fstConfiguration = FSTConfiguration.createDefaultConfiguration();
		fstConfiguration.setClassLoader(classLoader);
	}

	@Override
	public Object deserialize(InputStream inputStream) throws IOException {
		try{
			return fstConfiguration.getObjectInput(inputStream).readObject();
		}
		catch (ClassNotFoundException ex) {
			throw new NestedIOException("Failed to deserialize object type", ex);
		}
	}

	@Override
	public void serialize(Object object, OutputStream outputStream) throws IOException {
		// Do not close fstObjectOutput - that would prevent reuse and cause an error
		// see https://github.com/RuedigerMoeller/fast-serialization/wiki/Serialization
		@SuppressWarnings("resource")
		final FSTObjectOutput fstObjectOutput = fstConfiguration.getObjectOutput(outputStream);
		fstObjectOutput.writeObject(object);
		fstObjectOutput.flush();
	}
}

Using Kryo with Spring Session JDBC

Add Kryo as dependency to the project. For example, using Maven:

<dependency>
	<groupId>com.esotericsoftware</groupId>
	<artifactId>kryo</artifactId>
	<version>5.0.0-RC4</version>
</dependency>

And these add these classes:

KryoSessionConfig.java

import org.springframework.beans.factory.BeanClassLoaderAware;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.convert.ConversionService;
import org.springframework.core.convert.support.GenericConversionService;
import org.springframework.core.serializer.support.DeserializingConverter;
import org.springframework.core.serializer.support.SerializingConverter;

@Configuration
public class KryoSessionConfig implements BeanClassLoaderAware {

	private ClassLoader classLoader;

	@Bean
	public ConversionService springSessionConversionService() {
		final KryoDeserializerSerializer kryoDeserializerSerializer = new KryoDeserializerSerializer(classLoader);

		final GenericConversionService conversionService = new GenericConversionService();
		conversionService.addConverter(Object.class, byte[].class,
				new SerializingConverter(kryoDeserializerSerializer));
		conversionService.addConverter(byte[].class, Object.class,
				new DeserializingConverter(kryoDeserializerSerializer));
		return conversionService;
	}

	@Override
	public void setBeanClassLoader(final ClassLoader classLoader) {
		this.classLoader = classLoader;
	}
}

KryoDeserializerSerializer.java

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.springframework.core.serializer.Deserializer;
import org.springframework.core.serializer.Serializer;

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.Output;
import com.esotericsoftware.kryo.util.Pool;

public class KryoDeserializerSerializer implements Serializer<Object>, Deserializer<Object> {
	private final ClassLoader classLoader;
	
	// Pool constructor arguments: thread safe, soft references, maximum capacity
	private final Pool<Kryo> kryoPool = new Pool<Kryo>(true, true) {
	   protected Kryo create () {
	      final Kryo kryo = new Kryo();
	      kryo.setClassLoader(classLoader);
	      kryo.setRegistrationRequired(false);
	      return kryo;
	   }
	};
	
	public KryoDeserializerSerializer(final ClassLoader classLoader) {
		this.classLoader = classLoader;
	}

	@Override
	public Object deserialize(InputStream inputStream) throws IOException {
		final Kryo kryo = kryoPool.obtain();
		try(final Input input = new Input(inputStream)){
			return kryo.readObjectOrNull(input, null);
		}finally {
			kryoPool.free(kryo);
		}
	}

	@Override
	public void serialize(Object object, OutputStream outputStream) throws IOException {
		final Kryo kryo = kryoPool.obtain();
		try(final Output output = new Output(outputStream)){
			kryo.writeObject(output, object);
		}finally {
			kryoPool.free(kryo);
		}
	}
}

How to Choose which Serializer to Use

The process of selecting which serializer to use for an application should be done only through testing, both for functionality and for performance.

For some applications, a serializer won’t work due to known limitations in the serializer or bugs in it. For example, FST doesn’t currently support the Java 8 time classes, so if your application stores session data using such a class, FST is not for you. With Kryo, I ran into a bug stopping me from using it (which will be fixed in version 5.0.0-RC5 and later).

Performance will also vary between serializers for each application. Factors that impact performance include exactly what is being serialized, how big that data is, how often it’s accessed, the version of Java, and how the system is configured. FST has published some benchmarks, but that information must be taken with a grain of salt as those benchmarks are only measuring very specific, isolated scenarios. That data does provide general guidance though – you can expect better performance when you switch from the Java serializer to FST, for example, but testing of the full application will need to be done to determine if the improvement is 0.1% or 10%.

Categories: Uncategorized Tags:

Performance Testing WebDAV Clients

March 18th, 2019 No comments

Part of migrating applications from on-premises hosting to cloud hosting (AWS, Azure, etc) involves re-evaluating how users access their data. A recent migration involved users running Windows 10 accessing a Windows file share using the SMB protocol. Since SMB isn’t safe to run directly over the Internet (it’s usually not encrypted and it has a long history of security vulnerabilities), potential options included tunneling SMB within a VPN or changing away from the SMB protocol. One such alternative is WebDAV.

Web Distributed Authoring and Versioning (WebDAV) provides the same basic functionality as NFS, SMB, or FTP over the familiar, widely deployed well supported HTTP protocol. It was literally designed for this use case.

Testing Methodology

I evaluated 7 approaches for clients to access a WebDAV share, testing the performance of each and noting pros and cons.

Testing was performed in March 2019. The same client, same server, and same connection was used for each test.

The client is a Dell Latitude 7490 running Fedora 29. Windows 10 (version 1809 build 17763.348) was run as a VM in KVM using user mode networking to connect to the Internet. For the smb+davfs2 testing, smb+davfs2 ran in Docker running in Linux and the Windows 10 VM connected to it.

The WebDAV server is running Apache 2.4.38 and Linux kernel 5.0.0-rc8. The WebDAV site is served over HTTPS.

Ping time between them is 18ms average, httping to the WebDAV server URL is 140ms average.

Windows 10’s Redirector over HTTP/1.1

WebDAV Redirector is the name of the built in WebDAV client in Windows. It’s accessed from Windows Explorer by using the “Map network drive…” option and selecting “Connect to a web site that you can use to store your documents and pictures”

Recent versions of Windows 10 support HTTP/2, so for this test, the “h2” protocol was disabled on the Apache server.

Windows 10’s Redirector over HTTP/2

This test is the same as the previous one except the “h2” protocol is enabled on the Apache server.

WebDrive

WebDrive (version 2019 build 5305) was used with “Basic Cache Settings” set to “Multi-User.” All other settings were left at their defaults.

WebDrive does not currently support TLS 1.3 or HTTP/2; I suspect these would improve performance.

davfs2 on Linux

davfs2 is a Linux file system driver which allows mounting WebDAV shares just like any other file system.

For this testing, the client is Linux, not Windows 10. Therefore, this test isn’t a real solution (the clients do need to run Windows 10). This test is for comparison and testing on component of a possible solution (Samba + davfs).

davfs2 does not support HTTP/2.

davfs2 offer a number of configuration options; here’s what was used for this testing:

/home/candrews/.davfs2/davfs2.conf
[/home/candrews/webdavpoc]
use_locks 1
delay_upload 0 # disable write cache so users know that when the UI tells them the upload is done it’s actually done

And the mount command:

sudo mount -t davfs https://www.integralblue.com/cacdav /home/candrews/webdavpoc/ -ouid=1000,gid=1000,conf=/home/candrews/.davfs2/davfs2.conf

The davfs2 version is 1.5.4.

davfs2 with locks disabled (using a Linux client)

This test is the same as the previous one but this WebDAV locks disabled by setting “use_locks 1” in davfs2.conf.

Samba + davfs2

This test uses Samba to expose the davfs2 mount from the davfs2 test and uses Windows 10 to access the result SMB share.

The davfs2 mount exposed using SMB with the dperson/samba docker image using:

name smb -e USERID=1000 -e GROUPID=1000 --security-opt label:disable -v /home/candrews/webdavpoc:/webdavpoc:Z --rm -it -p 139:139 -p 445:445 -d dperson/samba -s "webdavpoc;/webdavpoc;yes;no;no;testuser" -u "testuser;testpass" -g "vfs objects ="

The biggest challenge with this approach, which I didn’t address, is with regards to security. The davfs2 mount that Samba exposes uses fixed credentials – so all Windows 10 users using the smb+davfs proxy will appear to the WebDAV server to be the same user. There is no way to pass through the credentials from the Windows 10 client through Samba through davfs2 to the WebDAV server. I imagine this limitation may disqualify this solution.

samba + davfs2 with locks disabled

This test is the same as the previous one but WebDAV locks are disabled in davfs2.

Results

Conclusion

Key takeaways from the results:

  • The built in Windows Redirector (which is free and the easiest solution) is by far the slowest.
  • WebDAV benefits greatly from HTTP/2. HTTP/2 was designed to improve the performance of multiple concurrent small requests (as is the case of HTML with references to CSS, Javascript, and images), and that’s exactly how WebDAV works. Each download is a PROPFIND and a GET, and each upload is a PROPFIND, PROPPUT, PUT, and, if locking is enabled, LOCK and UNLOCK.
  • Disabling WebDAV locks improves performance. By eliminating the LOCK/UNLOCK requests for every file, upload performance doubled. Unfortunately, disabling WebDAV locks for the built in Windows Redirector requires a registry change, a non-starter for many organizations. WebDrive has locks disabled by default. davfs2 only uses locks when editing files (when a program actually has the file open for writing), not when uploading a new file.
  • Surprisingly, the overall fastest (by far) approach involves a middle box running Linux using smb+davfs2. Adding a hop and another protocol usually slows things down, but not in this case.
  • davfs2 is fast because it opens more HTTP connections allowing it to do more requests in parallel. WebDrive supports adjusting the concurrency too; in “Connection Settings” there are settings for “Active Connections Limit” (defaults to 4) and “Active Upload Limit” (defaults to 2). Adjusting these settings would impact performance.
  • The client/server connection was high bandwidth (~100mpbs) and low latency (~18ms). Lower bandwidth and high latency would result in even greater benefits from using HTTP/2 and disabling locking.

Considerations for future research / testing include:

  • AWS Storage Gateway may be a solution. Concerns for this approach include:
    • Incompatible with MITM HTTPS-decrypting security devices
    • Uses a local, on premises cache (called an “upload buffer” in the documentation) so changes are not immediately available globally
    • Integrates with Active Directory for authentication/authorization, but setting up for a large number of users is onerous.
    • Requires the use of S3 for file storage, which may not be compatible with the servers in AWS
  • Various other WebDAV clients for Windows, such as DokanCloudFS, NetDrive, DirectNet Drive, ExpanDrive, and Cyberduck / Mountain Duck.
  • Further tweaks to configuration options, such as the concurrency settings in WebDrive and davfs2
Categories: Uncategorized Tags:

The Sad Story of TCP Fast Open

March 13th, 2019 No comments

If there’s a way to make something fast, you’ve got my attention. Especially when there’s a way to make a lot of things fast with a simple change – and that’s what TCP Fast Open (TFO) promises to do.

TFO (RFC 7413) started out in 2011 as a way to eliminate one of the round trips involved in opening a TCP connection. Since TCP (Transmission Control Protocol) is the underlying fundamental technology used for almost all connections on the Internet, an improvement to TCP connection establishment performance would improve the performance of virtually everything, especially web browsing. In early testing discussed at the 2011 Linux Plumbers Conference, Google found that TFO reduced page load times by 4-40%. Because of the elimination of a round trip, the slowest, highest latency connections would benefit the most – TFO promised to be a great improvement for many users.

Given the immediate success, support for this performance improving technology rapidly grew. In 2012, Linux 3.7 gained support for client and server TFO. In 2013, Android gained support when KitKat (4.4) was released using the Linux 3.10 kernel. In 2015, iOS gained support. In 2016, Windows 10 got support in the Anniversary Update. Even load balancers, such as F5, added support.

And yet, today, not one browser supports it. Chrome, Firefox, and Edge all have use of TFO disabled by default.

So, what happened to this technology that once sounded so promising?

Initial Optimism Meets Hard Reality

I attribute the failure to achieve widespread adoption of TCP Fast Open to four factors:

  1. Imperfect initial planning
  2. Middleboxes
  3. Tracking concerns
  4. Other performance improvements

Factor 1: Imperfect Initial Planning

TCP Fast Open was in trouble from its initial conception. Because it involves change to an operating system, it had to be done perfectly from the very beginning. Operating systems have long lifespans – updates happen slowly, backwards compatibility is paramount, and changes are, rightfully so, difficult to make. So, when the TFO specification wasn’t perfect the first time, it was a major blow to the changes of its ever achieving widespread adoption.

TFO requires the allocation of a new, dedicated TCP Option Kind Number. Option Kind Numbers specify which features are in use during a TCP connection and, as a new feature, TFO requires a new TCP Option Kind Number. Since TFO was experimental when it started out, it used a number (254 with magic 0xF989) from the experimental allocation (as described in RFC 4727). This was quickly ingrained into Windows, iOS, Linux. and more. As the saying goes, “nothing is as permanent as a temporary solution.”

So, when TFO left experimental status with RFC 7413, they released a statement saying that all current versions should migrated over to the new option. Or, in more complex terms,“Existing implementations that are using experimental option 254 per [RFC6994] with magic number 0xF989 (16 bits) as allocated in the IANA “TCP Experimental Option Experiment Identifiers (TCP ExIDs)” registry by this document, should migrate to use this new option (34) by default.”

Did all implementations migrate? If they did, they would lose compatibility with those that didn’t migrate. Therefore, all systems must now support both the experimental TCP Option Kind Number and the permanent one.

This issue isn’t a deal breaker – but it certainly wasn’t a great way to start out.

Factor 2: Middleboxes

Middleboxes are the appliances that sit between the end user and the server they’re trying to reach. They’re firewall, proxies, routers, caches, security devices, and more. They’re rarely updated, very expensive, and run proprietary software. Middleboxes are, in short, why almost everything runs over HTTP today and not other protocols as the original design for the Internet envisioned.

The first sign of trouble appeared in the initial report from Google in 2011 regarding TFO. As reported by LWN, “about 5% of the systems on the net will drop SYN packets containing unknown options or data. There is little to be done in this situation; TCP fast open simply will not work. The client must thus remember cases where the fast-open SYN packet did not get through and just use ordinary opens in the future.”

Over the years, Google and Mozilla did more testing and eventually found that TFO wasn’t beneficial. Clients that initiate TFO connections encounter failures requiring them to re-try without TFO so often that, on average, TFO costs more time than it saves. In some networks, TFO never works – for example, China Mobile’s firewall consistently fails to accept TFO requiring every connection to be retried without it, leading to TFO actually increasing roundtrips.

Middleboxes are probably the fatal blow for TFO: the existing devices won’t be replaced for (many) years, and the new replacement devices may have the same problems.

Factor 3: Tracking Concerns

When a client makes an initial connection to a host, TFO negotiates a unique random number called a cookie; on subsequent connections to the same host, the client uses the cookie to eliminate one round trip. Using this unique cookie allows servers using TFO to track users. For example, if a user browses to a site, then opens an incognito window and goes to the same site, the same TFO cookie would be used in both windows. Furthermore, if a user goes to a site at work, then uses the same browser to visit that site from a coffee shop, the same TFO cookie would be used in both cases, allowing the site to know it’s the same user.

In 2011, tracking by the governments and corporations wasn’t nearly as much of a concern as it is today. It would still be two years before Edward Snowden would release documents describing the US government’s massive surveillance programs.

But, in 2019, tracking concerns are real. TFO potential to be used for user tracking makes it unacceptable for most use cases.

One way to mitigate tracking concerns would be for the TFO cookie cache to be cleared whenever the active network changes. Windows/Linux/MacOS/FreeBSD/etc should consider clearing the OS’s TFO cookie cache when changing networks. See this discussion on curl’s issue tracker for more.

Factor 4: Other Performance Improvements

When TFO started out, HTTP/2 was not yet in use – in fact, HTTP/2’s precursor, SPDY, had a draft until 2012. With HTTP/1, a client would make many connections to the same server to make parallel requests. With HTTP/2, clients can make parallel requests over the same TCP connections. Therefore, since it setups up far fewer TCP connections, HTTP/2 benefits much less than HTTP/1 from TFO.

HTTP/3 plans to use UDP (User Datagram Protocol) to reduce connection setup round trips gaining the same performance advantage of TFO but without its problems. UDP is a fundamental Internet protocol like TCP but originally designed for one way connectionless communication. HTTP/3 will build a highly optimized connection establishment system logically analogous to TCP on top of UDP. The end result will be faster than what TCP can do even with TFO.

TLS (Transport Layer Security) 1.3 offers another improvement that reduces round trips called 0RTT. TLS is the system used for encryption in HTTPS so improvements to TLS potentially improve all users of HTTPS.

In the end, performance has been improving without requiring TFO’s drawbacks/costs.

The Future of TFO

TFO may never be universally used, but it still has its place. The best use case for TFO is with relatively new clients and servers, connected by a network using either no middleboxes or only middleboxes that don’t interfere with TFO, when user tracking isn’t a concern.

Domain Name System (DNS) is such a use case. DNS is how software (such as a browser) resolves human-readable names (such as integralblue.com) to an IP (Internet Protocol) address to which the computer can connect. DNS is very latency sensitive and very frequently used – eliminating the latency from one round trip would give a perceivable improvement to users. The same TCP connections are made from the same clients to the same servers repeatedly, which is TFO’s best case scenario. And, there’s no tracking concern since many DNS clients and servers don’t move around (there’s no “incognito” mode for DNS). Stubby, Unbound, dnsmasq, BIND, and PowerDNS, for example, include or are currently working on support for TFO.

TFO is already supported by all the major operating systems so it is here to stay. The question is, will it ever see widespread real world adoption? In the short term, TFO will be adopted in specialty areas such as DNS. But, eventually, , perhaps in the 5 or 10 year time frame, TFO will see widespread adoption on the Web as troublesome middleboxes are gradually replaced allowing browsers to enable support for it. By that time, however, HTTP/3 will be widely deployed, offering a better performance improvement than TFO could ever offer for the use case of web browsing. In the end, TFO is an interesting example of an idea and its implementation being challenged (due to technical limitations and a changing political landscape), resulting in its inability to live up to expectations. However, like so many other technologies, the original implementation isn’t what matters most – the idea is the significant part. In the case of TFO, the performance improving concept behind it will benefit users for years to it influences future technologies such as HTTP/3.

Categories: Uncategorized Tags:

WordPress on AWS The Easy Way with VersionPress

January 23rd, 2019 No comments

Developing and administering WordPress can be painful, especially when requirements include the ability to have a team of developers, the ability for a developer to run the site (including content) on their system (so they can reproduce issues caused by content or configuration), multiple content authors, multiple environments (such as staging and production), and the ability to roll back any problematic changes (content, code, or configuration) with ease.

To solve this problem, I developed a solution to make things easier: VersionPress on AWS. It uses Docker to allow developers to run the site locally and an assortment of AWS services to host the site. Check it out and I think you’ll find that life becomes a little bit easier.

VersionPress on AWS is Free Software under the GPLv3 hosted at gitlab. Contribution in the forms of issue reports and pull request are very much welcome.

What is VersionPress?

VersionPress stores content (posts, media, etc) as well as code (themes, plugins, configuration, etc) in source control (git).

  • By looking at the git log, it’s quick and easy to see who changed what and when.
  • All code (plugins, themes, WordPress core itself) and content (pages, posts, comments, configuration) are stored in git. This approach allows content changes as well as code changes can be reverted if there’s a problem and merged between branches for different environments.
  • Wipe out and recreate the environment at any time without data loss – everything is in git. No need to worry about the AWS RDS server. Migrate between RDS for MariaDB and Aurora at any time.
  • Need a staging site, or a new site to test work in progress? Create a new branch and launch a new stack, be up and running in minutes
  • Run the exact same site with the same content locally so you can reproduce issues in production effortlessly – no more “works on my machine” situations

Hosting with AWS

Need a small, cheap staging site, but also a full fledged scalable production site with a CDN? Use the same stack for both – simply specify different parameter values. Change parameter values whenever you want without downtime or data loss. For example, when starting out, leave the CloudFront CDN off to save money. When the site becomes popular, add the CloudFront CDN to better handle the load and improve performance for end users.

AWS features leveraged include:

Docker

Docker is used to run WordPress in AWS Elastic Beanstalk as well as for developers running the site locally. This consistency reduces the occurrences of “it works on my machine” situations and gets new developers on-boarded quicker.

When not to use VersionPress on AWS

Since VersionPress commits all content changes to git, content changes are a bit slower. Therefore, if the site is very content change heavy, such as if it’s a forum with many frequent comments being made, VersionPress on AWS may not be the right solution.

However, the vast majority of WordPress sites have very infrequent content changes, so the slightly slower writes are rarely an issue.

Get Started

Check out the VersionPress on AWS documentation to get started.

Categories: Uncategorized Tags:

DNSSEC on OpenWrt 18.06

August 10th, 2018 10 comments
DNSSEC ensures that the results of DNS queries (for DNSSEC enabled domains) are authentic. For example, integralblue.com uses DNSSEC, so if an attacker (using a man in the middle or spoofing attack) changes the IP address that www.integralblue.com resolves to, then a DNS resolver supporting DNSSEC will be able to tell and return an error.

DNSSEC provides authentication and integrity; it does not provide for confidentiality. For confidentiality (so your ISP, for example, cannot tell what DNS queries are being made), you can easily add TLS over DNS which I’ve described how to do in OpenWrt in another post.

By setting up DNSSEC on your OpenWrt router, you protect your entire network as all clients will perform DNS requests using your OpenWrt router’s DNS server which in turn will do DNSSEC checking for all queries.

Setting up DNSSEC on OpenWrt 18.06 is remarkably easy. You can use the LuCI web interface to perform these steps or shell command over ssh; I’m providing the commands here.

  1. Refresh the package list: opkg update
  2. Swap dnsmasq for dnsmasq-full (-full includes DNSSEC support) and remove odhcpd-ipv6only: opkg install dnsmasq-full --download-only && opkg remove dnsmasq odhcpd-ipv6only && opkg install dnsmasq-full --cache . && rm *.ipk
  3. Edit /etc/config/dhcp
    In the config dnsmasq section, add (or change the values of, if these settings already exist) these settings:

    • option dnssec '1'
    • option dnsseccheckunsigned '1'
  4. Restart dnsmasq so the changes take effect: /etc/init.d/dnsmasq restart

Enjoy knowing that now no one is tampering with your DNS queries.


Categories: Uncategorized Tags:

DNS Over TLS on OpenWrt 18.06

August 10th, 2018 16 comments
DNS over TLS encrypts DNS queries so no one between you and the DNS server you’re using (which, by default using these steps, will be Cloudflare’s 1.1.1.1), can tell what DNS queries/responses are being exchanged.

DNS over TLS provides confidentiality but not integrity or authenticity. For those, you need to setup DNSSEC which I’ve described how to do on OpenWrt in another post.

By setting up DNS over TLS on your OpenWrt router, you protect your entire network as all clients will perform DNS requests using your OpenWrt router’s DNS server which in turn will use DNS over TLS to perform the actual resolution.

Setting up DNS over TLS using Stubby on OpenWrt 18.06 is remarkably easy. You can use the LuCI web interface to perform these steps or shell command over ssh; I’m providing the commands here.

  1. Refresh the package list: opkg update
  2. Install the stubby package: opkg install stubby
  3. Start stubby: /etc/init.d/stubby start
  4. Set stubby to start automatically at boot: /etc/init.d/stubby enable
  5. Use stubby as the DNS server by editing /etc/config/dhcp
    In the config dnsmasq section, add (or change the values of, if these settings already exist) these settings:

    • option noresolv '1'
    • list server '127.0.0.1#5453'
  6. Restart dnsmasq so the changes take effect: /etc/init.d/dnsmasq restart

If you’d rather use a different DNS over TLS server than Cloudflare’s 1.1.1.1, edit /etc/stubby/stubby.yml.

Now you can restart assured that your DNS queries can’t be seen by 3rd parties.

Categories: Uncategorized Tags:

MaybeGZIPInputStream

May 29th, 2018 No comments
I’m currently working on an application that persists Java serialized data (using ObjectOutputStream) in a database. Java’s serialization format compresses very well – so why not compress the data when storing it then decompress it while reading for a quick win? The problem is that there will still be legacy, uncompressed data, which the application will not be able to access if it assumes all data is now gzipped.

The solution is to use MaybeGZIPInputStream instead of GZIPInputStream. For example, when reading, instead of:

ObjectInputStream ois = new ObjectInputStream(new GZIPInputStream(databaseInputStream));

use MaybeGZIPInputStream instead:

ObjectInputStream ois = new ObjectInputStream(new MaybeGZIPInputStream(databaseInputStream));

And always write data using GZIPOutputStream. Now all of that existing data can be still be read, and newly written data gets the benefit of taking up much less storage (and taking up far less bandwidth / time being transferred between the application servers and the database).

Here’s the source code of MaybeGZIPInputStream:

import java.io.IOException;
import java.io.InputStream;
import java.io.PushbackInputStream;
import java.util.zip.GZIPInputStream;

/** Detect if the given {@link InputStream} contains compressed data. If it does, wrap it in a {@link GZIPInputStream}. If it doesn’t, don’t.
* @author Craig Andrews
*
*/
public class MaybeGZIPInputStream extends InputStream {

private final InputStream in;

public MaybeGZIPInputStream(final InputStream in) throws IOException {
final PushbackInputStream pushbackInputStream = new PushbackInputStream(in, 2);
if(isGZIP(pushbackInputStream)) {
this.in = new GZIPInputStream(pushbackInputStream);
}else {
this.in = pushbackInputStream;
}
}

private boolean isGZIP(final PushbackInputStream pushbackInputStream) throws IOException {
final byte[] bytes = new byte[2];
final int bytesRead = pushbackInputStream.read(bytes);
if(bytesRead > 0) {
pushbackInputStream.unread(bytes, 0, bytesRead);
}
if(bytesRead == 2) {
if ((bytes[0] == (byte) (GZIPInputStream.GZIP_MAGIC)) && (bytes[1] == (byte) (GZIPInputStream.GZIP_MAGIC >> 8))){
return true;
}
}
return false;
}

public int read() throws IOException {
return in.read();
}

public int hashCode() {
return in.hashCode();
}

public int read(byte[] b) throws IOException {
return in.read(b);
}

public boolean equals(Object obj) {
return in.equals(obj);
}

public int read(byte[] b, int off, int len) throws IOException {
return in.read(b, off, len);
}

public long skip(long n) throws IOException {
return in.skip(n);
}

public String toString() {
return in.toString();
}

public int available() throws IOException {
return in.available();
}

public void close() throws IOException {
in.close();
}

public void mark(int readlimit) {
in.mark(readlimit);
}

public void reset() throws IOException {
in.reset();
}

public boolean markSupported() {
return in.markSupported();
}

}

Categories: Uncategorized Tags:

SQS JMS Resource Adapter

May 7th, 2018 No comments
The recently released SQS JMS Resource Adapter allows JEE applications (running on any JEE application server, including Glassfish, Payara, JBoss, IBM Liberty, etc) to easily use AWS SQS as a JMS implementation. This resource adapter can be helpful in many situations, such as:

  • Migrating an existing JEE application from another JMS implementation (such as RabbitMQ, ActiveMQ, IBM MQ, etc) to AWS SQS.
  • Allowing the JMS implementation to be switched out. For example, developers can use the ActiveMQ resource adapter, and in production, this AWS SQS resource adapter could be used.

Grab the resource adapter from Maven Central and submit issues and pull requests over at GitHub.

Categories: Uncategorized Tags:

Trusting DoD Certificates in Docker and Beanstalk

May 1st, 2018 No comments

The US DoD (Department of Defense) uses its own root certificate when signing https certificates for its domains. For example, https://www.my.af.mil/ uses such a certificate. These root certificates are not trusted by any (commercial/public) operating system, browser, or other client. Therefore, in order to access these sites and not get an error, the DoD certificates must be trusted.

On Windows, go to DISA’s PKI and PKE Tools page and under “Trust Store” follow the directions for the “InstallRoot X: NIPR Windows Installer”

On Linux, download the certificates from MilitaryCAC’s Linux Information page (direct link to the certificates). Then follow your distribution’s instructions on how to install certificates to the trust store. For example, on Red Hat / CentOS / Fedora / Amazon Linux, copy the certificates to /etc/pki/ca-trust/source/anchors/ then run update-ca-trust. On Debian / Ubuntu and Gentoo, copy the certificates to /usr/local/share/ca-certificates/ then run update-ca-certificates.

On Docker, for a Red Hat / CentOS / Fedora / Amazon Linux (or other Fedora-type system) derived container, add the following to the Dockerfile:

#!/bin/bash
set -e # stop on all errors
RUN yum -y install openssl \
&& CERT_BUNDLE="Certificates_PKCS7_v5.3_DoD" \
&& curl "https://iasecontent.disa.mil/pki-pke/${CERT_BUNDLE}.zip" --output certs.zip \
&& unzip certs.zip "${CERT_BUNDLE}/${CERT_BUNDLE}.pem.p7b" \
&& openssl pkcs7 -in "${CERT_BUNDLE}/${CERT_BUNDLE}.pem.p7b" -print_certs -out "/etc/pki/ca-trust/source/anchors/${CERT_BUNDLE}.pem" \
&& update-ca-trust \
&& update-ca-trust force-enable \
&& rm -rf certs.zip "${CERT_BUNDLE}" \
&& yum -y remove openssl \
&& rm -rf /var/cache/yum

On AWS Elastic Beanstalk the .ebextensions mechanism can be used. In the jar/war/etc deployment archive, add these files:

.ebextensions/install_dod_certificates.config

packages:
  yum:
    bash: []
    curl: []
    openssl: []
    unzip: []
files:
  "/tmp/install_dod_certificates.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      set -Eeuo pipefail # stop on all errors
      cd /tmp
      CERT_BUNDLE="Certificates_PKCS7_v5.3_DoD"
      curl "https://iasecontent.disa.mil/pki-pke/${CERT_BUNDLE}.zip" --output certs.zip
      unzip certs.zip "${CERT_BUNDLE}/${CERT_BUNDLE}.pem.p7b"
      openssl pkcs7 -in "${CERT_BUNDLE}/${CERT_BUNDLE}.pem.p7b" -print_certs -out "/etc/pki/ca-trust/source/anchors/${CERT_BUNDLE}.pem"
      update-ca-trust
      update-ca-trust force-enable
      rm -rf certs.zip "${CERT_BUNDLE}"
commands:
  01install_dod_certificates:
    command: "/tmp/install_dod_certificates.sh"
Categories: Uncategorized Tags: