Spring ID to Entity Conversion

July 22nd, 2015 No comments

When using Spring with Hibernate or JPA, it can be very convenient to use objects as method parameters and automatically convert primary keys to those objects. For example:

  1. @RequestMapping(value = "/person/{personId}", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
  2. @ResponseBody
  3. public PersonDto getById(@PathVariable Person person) {
  4. 	return person;
  5. }

instead of:

  1. @RequestMapping(value = "/person/{personId}", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
  2. @ResponseBody
  3. public PersonDto getById(@PathVariable String personId) {
  4. 	return personService.getById(personId);
  5. }

Using this approach throughout a project (in controllers, services, etc) can lead to much more readable, shorter code.

Spring Data JPA provides a similar feature via DomainClassConverter. In my case, Spring Data wasn’t an option.

Here’s how to make this happen. First, set up a @Configuration class:

  1. package com.integralblue.candrews.config;
  3. import javax.annotation.PostConstruct;
  5. import org.springframework.context.annotation.Bean;
  6. import org.springframework.context.annotation.Configuration;
  7. import org.springframework.web.servlet.config.annotation.WebMvcConfigurationSupport;
  9. import com.integralblue.candrews.converter.EntityToIdConverter;
  10. import com.integralblue.candrews.converter.IdToEntityConverter;
  12. @Configuration
  13. public class WebMvcConfig extends WebMvcConfigurationSupport {
  14. 	@Bean
  15. 	protected IdToEntityConverter idToEntityConverter() {
  16. 		return new IdToEntityConverter();
  17. 	}
  19. 	@Bean
  20. 	protected EntityToIdConverter entityToIdConverter() {
  21. 		return new EntityToIdConverter();
  22. 	}
  24. 	@PostConstruct
  25. 	public void addConverters() {
  26. 		mvcConversionService().addConverter(idToEntityConverter());
  27. 		mvcConversionService().addConverter(entityToIdConverter());
  28. 	}
  29. }

Now add these 2 classes:

  1. package com.integralblue.candrews.converter;
  3. import java.util.Collections;
  4. import java.util.HashSet;
  5. import java.util.Set;
  7. import javax.persistence.EntityManager;
  8. import javax.persistence.PersistenceContext;
  9. import javax.persistence.metamodel.Attribute;
  10. import javax.persistence.metamodel.EntityType;
  11. import javax.persistence.metamodel.SingularAttribute;
  13. import org.springframework.beans.factory.annotation.Autowired;
  14. import org.springframework.core.convert.TypeDescriptor;
  15. import org.springframework.core.convert.converter.ConditionalGenericConverter;
  16. import org.springframework.format.support.FormattingConversionService;
  18. /**
  19.  * Converts an entity to its ID (the field in the entity annotated with
  20.  * {@link javax.persistence.Id}). This is similar to
  21.  * {@link org.springframework.data.repository.support.DomainClassConverter<T>}
  22.  * but since we don't use Spring Data, we can't use that.
  23.  */
  24. public class EntityToIdConverter implements ConditionalGenericConverter {
  26. 	@PersistenceContext
  27. 	private EntityManager entityManager;
  29. 	@Autowired
  30. 	private FormattingConversionService conversionService;
  32. 	@Override
  33. 	public Set<ConvertiblePair> getConvertibleTypes() {
  34. 		return Collections.singleton(new ConvertiblePair(Object.class,
  35. 				Object.class));
  36. 	}
  38. 	@SuppressWarnings("unchecked")
  39. 	@Override
  40. 	public boolean matches(TypeDescriptor sourceType, TypeDescriptor targetType) {
  41. 		EntityType<?> entityType;
  42. 		try {
  43. 			entityType = entityManager.getMetamodel().entity(
  44. 					sourceType.getType());
  45. 		} catch (IllegalArgumentException e) {
  46. 			return false;
  47. 		}
  48. 		if (entityType == null)
  49. 			return false;
  51. 		// In my opinion, this is probably a bug in Hibernate.
  52. 		// If the class has a @Id that is a complex type (such as another
  53. 		// entity), then entityType.getIdType() will throw an
  54. 		// IllegalStateException with the message "No supertype found"
  55. 		// To work around this, get the idClassAttributes, and if there is
  56. 		// exactly 1, use the java type of that.
  57. 		Class<?> idClass;
  58. 		if (entityType.hasSingleIdAttribute()) {
  59. 			idClass = entityType.getIdType().getJavaType();
  60. 		} else {
  61. 			Set<SingularAttribute<?, ?>> idAttributes = new HashSet<>();
  62. 			@SuppressWarnings("rawtypes")
  63. 			Set attributes = entityType.getAttributes();
  64. 			for (Attribute<?, ?> attribute : (Set<Attribute<?, ?>>) attributes) {
  65. 				if (attribute instanceof SingularAttribute<?, ?>) {
  66. 					SingularAttribute<?, ?> singularAttribute = (SingularAttribute<?, ?>) attribute;
  67. 					if (singularAttribute.isId()) {
  68. 						idAttributes.add(singularAttribute);
  69. 					}
  70. 				}
  72. 			}
  73. 			if (idAttributes.size() == 1) {
  74. 				idClass = idAttributes.iterator().next().getJavaType();
  75. 			} else {
  76. 				return false;
  77. 			}
  78. 		}
  80. 		return conversionService.canConvert(TypeDescriptor.valueOf(idClass),
  81. 				targetType);
  82. 	}
  84. 	@Override
  85. 	public Object convert(Object source, TypeDescriptor sourceType,
  86. 			TypeDescriptor targetType) {
  87. 		if (sourceType.getType() == targetType.getType()) {
  88. 			return source;
  89. 		}
  90. 		if (source == null) {
  91. 			return null;
  92. 		}
  93. 		EntityType<?> entityType = entityManager.getMetamodel().entity(
  94. 				sourceType.getType());
  95. 		Object ret = entityManager.getEntityManagerFactory()
  96. 				.getPersistenceUnitUtil().getIdentifier(source);
  97. 		if (ret == null)
  98. 			return null;
  99. 		return conversionService.convert(ret,
  100. 				TypeDescriptor.valueOf(entityType.getIdType().getJavaType()),
  101. 				targetType);
  102. 	}
  103. }
  1. package com.integralblue.candrews.converter;
  3. import java.util.Collections;
  4. import java.util.HashSet;
  5. import java.util.Set;
  7. import javax.persistence.EntityManager;
  8. import javax.persistence.NoResultException;
  9. import javax.persistence.PersistenceContext;
  10. import javax.persistence.metamodel.Attribute;
  11. import javax.persistence.metamodel.EntityType;
  12. import javax.persistence.metamodel.SingularAttribute;
  14. import org.apache.commons.lang.StringUtils;
  15. import org.springframework.beans.factory.annotation.Autowired;
  16. import org.springframework.core.convert.TypeDescriptor;
  17. import org.springframework.core.convert.converter.ConditionalGenericConverter;
  18. import org.springframework.format.support.FormattingConversionService;
  19. import org.springframework.security.access.prepost.PostAuthorize;
  21. /**
  22.  * Converts an ID (the field in the entity annotated with
  23.  * {@link javax.persistence.Id}) to the entity itself. This is similar to
  24.  * {@link org.springframework.data.repository.support.DomainClassConverter<T>}
  25.  * but since we don't use Spring Data, we can't use that.
  26.  */
  27. public class IdToEntityConverter implements ConditionalGenericConverter {
  29. 	@PersistenceContext
  30. 	private EntityManager entityManager;
  32. 	@Autowired
  33. 	private FormattingConversionService conversionService;
  35. 	@Override
  36. 	public Set<ConvertiblePair> getConvertibleTypes() {
  37. 		return Collections.singleton(new ConvertiblePair(Object.class,
  38. 				Object.class));
  39. 	}
  41. 	@SuppressWarnings("unchecked")
  42. 	@Override
  43. 	public boolean matches(TypeDescriptor sourceType, TypeDescriptor targetType) {
  44. 		EntityType<?> entityType;
  45. 		try {
  46. 			entityType = entityManager.getMetamodel().entity(
  47. 					targetType.getType());
  48. 		} catch (IllegalArgumentException e) {
  49. 			return false;
  50. 		}
  51. 		if (entityType == null)
  52. 			return false;
  54. 		// In my opinion, this is probably a bug in Hibernate.
  55. 		// If the class has a @Id that is a complex type (such as another
  56. 		// entity), then entityType.getIdType() will throw an
  57. 		// IllegalStateException with the message "No supertype found"
  58. 		// To work around this, get the idClassAttributes, and if there is
  59. 		// exactly 1, use the java type of that.
  60. 		Class<?> idClass;
  61. 		if (entityType.hasSingleIdAttribute()) {
  62. 			idClass = entityType.getIdType().getJavaType();
  63. 		} else {
  64. 			Set<SingularAttribute<?, ?>> idAttributes = new HashSet<>();
  65. 			@SuppressWarnings("rawtypes")
  66. 			Set attributes = entityType.getAttributes();
  67. 			for (Attribute<?, ?> attribute : (Set<Attribute<?, ?>>) attributes) {
  68. 				if (attribute instanceof SingularAttribute<?, ?>) {
  69. 					SingularAttribute<?, ?> singularAttribute = (SingularAttribute<?, ?>) attribute;
  70. 					if (singularAttribute.isId()) {
  71. 						idAttributes.add(singularAttribute);
  72. 					}
  73. 				}
  75. 			}
  76. 			if (idAttributes.size() == 1) {
  77. 				idClass = idAttributes.iterator().next().getJavaType();
  78. 			} else {
  79. 				return false;
  80. 			}
  81. 		}
  83. 		return conversionService.canConvert(sourceType,
  84. 				TypeDescriptor.valueOf(idClass));
  85. 	}
  87. 	@Override
  88. 	@PostAuthorize("hasPermission(returnObject, 'view')")
  89. 	public Object convert(Object source, TypeDescriptor sourceType,
  90. 			TypeDescriptor targetType) {
  91. 		if (sourceType.getType() == targetType.getType()) {
  92. 			return source;
  93. 		}
  94. 		EntityType<?> entityType = entityManager.getMetamodel().entity(
  95. 				targetType.getType());
  96. 		Object id = conversionService.convert(source, sourceType,
  97. 				TypeDescriptor.valueOf(entityType.getIdType().getJavaType()));
  98. 		if (id == null
  99. 				|| (id instanceof String && StringUtils.isEmpty((String) id))) {
  100. 			return null;
  101. 		}
  102. 		Object ret = entityManager.find(targetType.getType(), id);
  103. 		if (ret == null) {
  104. 			throw new NoResultException(targetType.getType().getName()
  105. 					+ " with id '" + id + "' not found");
  106. 		}
  107. 		return ret;
  108. 	}
  109. }
Categories: Uncategorized Tags:

Issues using Ehcache with ARC as the Hibernate Cache

July 22nd, 2015 No comments

When using Ehcache’s ARC (Automatic Resource Control) on a cache acting as the Hibernate cache, these kinds of warning will likely appear:

WARN [pool-1-thread-1] [ehcache.pool.impl.DefaultSizeOfEngine] sizeOf The configured limit of 100 object references was reached while attempting to calculate the size of the object graph. This can be avoided by adding stop points with @IgnoreSizeOf annotations. Since the CacheManger or Cache elements maxDepthExceededBehavior is set to “abort”, the sizing operation has stopped and the reported cache size is not accurate. If performance degradation is NOT an issue at the configured limit, raise the limit value using the CacheManager or Cache elements maxDepth attribute. For more information, see the Ehcache configuration documentation.

To fix this problem, you need to tell ARC what parts of Hibernate objects shouldn’t be counted when calculating object sizes so ARC doesn’t get into an infinite loop (when it hits circular references) or try to calculate the size of the entire Hibernate object graph.

One way to solve this problem is to use ehcache-sizeofengine-hibernate. To do so, simply add the jar to your application’s class path and it will be automatically discovered and used by Ehcache. However, since ehcache-sizeofengine-hibernate isn’t in Maven Central, or any publicly accessible Maven repository, it’s not really useful. I requested that it be published to Central a while ago, but the project seems to be unmaintained – so a more maintainable way to solve this problem is required.

A system property needs to be set pointing to a file that lists the classes and fields that should be ignored by ARC. In my Spring @Configuation class for configuring ehcache, I have this block:

static {
System.setProperty(net.sf.ehcache.pool.impl.DefaultSizeOfEngine.USER_FILTER_RESOURCE, "net.sf.ehcache.sizeof.filter.properties");

But you can really do this anywhere (including simply adding “-Dnet.sf.ehcache.sizeof.filter=net.sf.ehcache.sizeof.filter.properties” to the Java command line).

Next, create a file named “net.sf.ehcache.sizeof.filter.properties” in the classpath (in a Maven project, the src/main/resources directory makes sense) with these contents:

Categories: Uncategorized Tags:

Issues using Ehcache with ARC as the Thymeleaf Cache

July 22nd, 2015 No comments

Ehcache’s ARC (Automatic Resource Control) is pretty great – it more or less automatically and optimally handles cache sizing/allocation. Using ARC, one can effectively say, “Here’s 2GB to allocate for caching – no more than that – manage the cache as best as possible.” Of course, more complex rules can also be done (on a per cache basis instead of globally, for instance) but that’s the general idea.

I’ve used ARC on my last couple of projects to great success. However, it’s not without a bit of complexity. ARC has to calculate the size of objects in the cache to determine how much space is used by the cache, and calculating sizes of Java objects isn’t easy or obvious. Besides difficult to do yet seemingly simple sound tasks as answering “How big is an object reference pointer?” there are object graph traversal problems, and those object graph issues are the ones that I’ve run into a few times in my ARC adventures.

When using Ehcache with ARC to store Thymeleaf templates (for example, when using Spring Cache abstraction backed by Ehcache as the Thymeleaf cache implementation as I previous discussed), you’ll run into messages like this one:

WARN [pool-1-thread-1] [ehcache.pool.impl.DefaultSizeOfEngine] sizeOf The configured limit of 100 object references was reached while attempting to calculate the size of the object graph. This can be avoided by adding stop points with @IgnoreSizeOf annotations. Since the CacheManger or Cache elements maxDepthExceededBehavior is set to “abort”, the sizing operation has stopped and the reported cache size is not accurate. If performance degradation is NOT an issue at the configured limit, raise the limit value using the CacheManager or Cache elements maxDepth attribute. For more information, see the Ehcache configuration documentation.

To fix this problem, you need to tell ARC what parts of Thymeleaf objects shouldn’t be counted when calculating object sizes so ARC doesn’t get into an infinite loop (when it hits circular references) or try to calculate the size of the entire Spring context.

A system property needs to be set pointing to a file that lists the classes and fields that should be ignored by ARC. In my Spring @Configuation class for configuring ehcache, I have this block:

static {
System.setProperty(net.sf.ehcache.pool.impl.DefaultSizeOfEngine.USER_FILTER_RESOURCE, "net.sf.ehcache.sizeof.filter.properties");

But you can really do this anywhere (including simply adding “-Dnet.sf.ehcache.sizeof.filter=net.sf.ehcache.sizeof.filter.properties” to the Java command line).

Next, create a file named “net.sf.ehcache.sizeof.filter.properties” in the classpath (in a Maven project, the src/main/resources directory makes sense) with these contents:


That’s it!

Note that I’ve requested that Thymeleaf make the org.thymeleaf.Template class serializable, which would remove the need to do anything (ARC would “Just Work” if this change is made): https://github.com/thymeleaf/thymeleaf/issues/378 So hopefully, with a future version of Thymeleaf, all of this information will be useless and nothing more than of historical interest.

Categories: Uncategorized Tags:

Using Spring Cache as the Thymeleaf Cache

July 22nd, 2015 No comments

Thymeleaf includes caching that can be used to cache templates, fragments, messages, and expressions. The default implementation is an in-memory cache using standard Java collections. Wouldn’t it be nice to use the Spring Cache Abstraction instead? The advantages of such a setup include:

  • Consistency – why have many separate caching implementations each configured in different ways when you can have one?
  • Abstraction – Spring Cache Abstraction can be easily configured to use any number of implementations, such as ehcache, Guava, JDK collections, or any JSR-107 provider
  • All the advantages of the implementation chosen – For example, if your cahce provider supports it, you get statistics, memory usage information, and more

To connect Thymeleaf (I’m using 2.1.x, but this should work for any version 2.1 or later) to Spring (I’m using 4.1.x, but this approach should work for 4.0 or later).

Here’s a working configuration:

  1. package com.integralblue.candrews;
  3. import java.util.List;
  4. import java.util.Properties;
  6. import org.springframework.beans.factory.annotation.Autowired;
  7. import org.springframework.beans.factory.annotation.Value;
  8. import org.springframework.cache.Cache;
  9. import org.springframework.cache.CacheManager;
  10. import org.springframework.context.MessageSource;
  11. import org.springframework.context.annotation.Bean;
  12. import org.springframework.context.annotation.Configuration;
  13. import org.thymeleaf.Template;
  14. import org.thymeleaf.cache.AbstractCacheManager;
  15. import org.thymeleaf.cache.ICache;
  16. import org.thymeleaf.cache.ICacheEntryValidityChecker;
  17. import org.thymeleaf.cache.ICacheManager;
  18. import org.thymeleaf.dom.Node;
  19. import org.thymeleaf.extras.springsecurity4.dialect.SpringSecurityDialect;
  20. import org.thymeleaf.spring4.SpringTemplateEngine;
  21. import org.thymeleaf.spring4.templateresolver.SpringResourceTemplateResolver;
  22. import org.thymeleaf.spring4.view.ThymeleafViewResolver;
  23. import org.thymeleaf.templateresolver.ITemplateResolver;
  24. import org.thymeleaf.templateresolver.TemplateResolver;
  26. @Configuration
  27. public class ThymeleafConfig {
  28. 	private static final String VIEWS = "/WEB-INF/views/";
  29. 	private static final String TEMPLATE_CACHE_NAME = "thymeleafTemplateCache";
  30. 	private static final String FRAGMENT_CACHE_NAME = "thymeleafFragmentCache";
  31. 	private static final String MESSAGE_CACHE_NAME = "thymeleafMessageCache";
  32. 	private static final String EXPRESSION_CACHE_NAME = "thymeleafExpressionCache";
  34. 	@Value("${thymeleaf.cache}")
  35. 	private boolean thymeleafCache;
  37. 	@Autowired
  38. 	private CacheManager cacheManager;
  40. 	@Autowired
  41. 	private MessageSource messageSource;
  43. 	@Bean
  44. 	public ThymeleafViewResolver viewResolver() {
  45. 		ThymeleafViewResolver viewResolver = new ThymeleafViewResolver();
  46. 		viewResolver.setCache(thymeleafCache);
  47. 		viewResolver.setTemplateEngine(templateEngine());
  48. 		viewResolver.setCharacterEncoding("UTF-8");
  49. 		return viewResolver;
  50. 	}
  52. 	@Bean
  53. 	public SpringTemplateEngine templateEngine() {
  54. 		SpringTemplateEngine templateEngine = new SpringTemplateEngine();
  55. 		templateEngine.setCacheManager(thymeleafCacheManager());
  56. 		templateEngine.setTemplateResolver(templateResolver());
  57. 		templateEngine.addDialect(new SpringSecurityDialect());
  58. 		templateEngine.setMessageSource(messageSource);
  59. 		return templateEngine;
  60. 	}
  62. 	@Bean
  63. 	public ITemplateResolver templateResolver() {
  64. 		TemplateResolver templateResolver = new SpringResourceTemplateResolver();
  65. 		templateResolver.setPrefix(VIEWS);
  66. 		templateResolver.setSuffix(".html");
  67. 		templateResolver.setTemplateMode("HTML5");
  68. 		templateResolver.setCacheable(thymeleafCache);
  69. 		return templateResolver;
  70. 	}
  72. 	@Bean
  73. 	ICacheManager thymeleafCacheManager() {
  74. 		return new AbstractCacheManager() {
  76. 			@Override
  77. 			protected ICache<String, Template> initializeTemplateCache() {
  78. 				return templateCache();
  79. 			}
  81. 			@Override
  82. 			protected ICache<String, List<Node>> initializeFragmentCache() {
  83. 				return fragmentCache();
  84. 			}
  86. 			@Override
  87. 			protected ICache<String, Properties> initializeMessageCache() {
  88. 				return messageCache();
  89. 			}
  91. 			@Override
  92. 			protected ICache<String, Object> initializeExpressionCache() {
  93. 				return expressionCache();
  94. 			}
  95. 		};
  96. 	}
  98. 	@Bean
  99. 	ICache<String, Template> templateCache() {
  100. 		return new CacheToICacheAdapter<Template>(
  101. 				cacheManager.getCache(TEMPLATE_CACHE_NAME));
  102. 	}
  104. 	@Bean
  105. 	ICache<String, List<Node>> fragmentCache() {
  106. 		return new CacheToICacheAdapter<List<Node>>(
  107. 				cacheManager.getCache(FRAGMENT_CACHE_NAME));
  108. 	}
  110. 	@Bean
  111. 	ICache<String, Properties> messageCache() {
  112. 		return new CacheToICacheAdapter<Properties>(
  113. 				cacheManager.getCache(MESSAGE_CACHE_NAME));
  114. 	}
  116. 	@Bean
  117. 	ICache<String, Object> expressionCache() {
  118. 		return new CacheToICacheAdapter<Object>(
  119. 				cacheManager.getCache(EXPRESSION_CACHE_NAME));
  120. 	}
  122. 	private static final class CacheToICacheAdapter<V> implements
  123. 			ICache<String, V> {
  124. 		private final Cache cache;
  126. 		private static final class CacheEntry<V> {
  127. 			private final V value;
  128. 			private final long timestamp;
  130. 			public CacheEntry(V value) {
  131. 				this.value = value;
  132. 				this.timestamp = System.currentTimeMillis();
  133. 			}
  135. 			public V getValue() {
  136. 				return value;
  137. 			}
  139. 			public long getTimestamp() {
  140. 				return timestamp;
  141. 			}
  142. 		}
  144. 		public CacheToICacheAdapter(Cache cache) {
  145. 			this.cache = cache;
  146. 		}
  148. 		@Override
  149. 		public void put(String key, V value) {
  150. 			cache.put(key, new CacheEntry<V>(value));
  151. 		}
  153. 		@Override
  154. 		public V get(String key) {
  155. 			@SuppressWarnings("unchecked")
  156. 			CacheEntry<V> cacheEntry = cache.get(key, CacheEntry.class);
  157. 			if (cacheEntry == null) {
  158. 				return null;
  159. 			} else {
  160. 				return cacheEntry.getValue();
  161. 			}
  162. 		}
  164. 		@Override
  165. 		public V get(
  166. 				String key,
  167. 				ICacheEntryValidityChecker<? super String, ? super V> validityChecker) {
  168. 			@SuppressWarnings("unchecked")
  169. 			CacheEntry<V> cacheEntry = cache.get(key, CacheEntry.class);
  170. 			if (cacheEntry == null) {
  171. 				return null;
  172. 			} else {
  173. 				V value = cacheEntry.getValue();
  174. 				if (validityChecker == null
  175. 						|| validityChecker.checkIsValueStillValid(key, value,
  176. 								cacheEntry.getTimestamp())) {
  177. 					return value;
  178. 				} else {
  179. 					return null;
  180. 				}
  181. 			}
  182. 		}
  184. 		@Override
  185. 		public void clear() {
  186. 			cache.clear();
  187. 		}
  189. 		@Override
  190. 		public void clearKey(String key) {
  191. 			cache.evict(key);
  192. 		}
  194. 	}
  195. }

I’ve requested that Thymeleaf include this feature as part of the Spring integration in Thymeleaf 3.0: https://github.com/thymeleaf/thymeleaf-spring/issues/89 So hopefully, with future versions of Thymeleaf, none of this extra work will be necessary and things will Just Work™.

If using Ehcache as the backing cache for Thymeleaf, and ARC is being used, errors may occur – see “Issues using Ehcache with ARC as the Thymeleaf Cache” for more information.

Categories: Uncategorized Tags:

Precompiling with Web Deploy

April 27th, 2012 5 comments

Web Deploy is a great tool for deploying ASP.NET projects: it simplifies deployments, can target IIS 6 or 7, is pretty easy to get working on the server and the project side, and is supported by many hosting providers. It works really well when used along with continuous integration. In the first phase, the project is built and a Web Deploy package is made; the command used is (for example) “msbuild  [something.sln or something.csproj or something.vbproj] /m /p:Configuration=release /p:DeployOnBuild=True /p:DeployTarget=Package /p:skipJsHint=true”. In the second phase, the Web Deploy package is deployed to servers for testing.

One of the problems I noticed was that a user could browse around the site and hit a compilation error. For instance, if there was a misspelled variable in an ASPX page, the user would get a standard “500” error page.

One of the purposes of continuous integration, and build/testing systems in general, is to avoid such problems. If the code behind fails to compile, the team is notified immediately – but if the ASPX page fails to compile, we have to wait for a user to report the problem. That is not acceptable.

The aspnet_compiler tool is the solution to this problem. If you use ASP.NET MVC, your project’s MSBuild file will contain a target named “MVCBuildViews” that will run aspnet_compiler for you. Enabling the use of this target by setting the “MvcBuildViews” property makes the build take a bit longer, but there are less surprises – to me, the benefit far outweighs the cost.

So the first option is to simply enable “MvcBuildViews.” “MVCBuildViews” writes the precompiled files to the ASP.NET temporary directory (the precompiled output is not included in the Web Deploy package), so they’ll be used only by the system that invoked the target (in my case, the continuous integration server). So for server deployments, you get the benefit of knowing that the pages compiled (solving the original problem), but you don’t get the other major benefit of precompiling – the startup performance boost.

It turns that combining aspnet_compiler and Web Deploy is a bit tricky. I won’t go into the details, other than to say I figured it out and made an MSBuild target file that can be easily included into projects that use Web Deploy.

So, to get Web Deploy working with precompilation:

  1. Grab Precompile.targets and put it into the directory with your .csproj/.vbproj project file.
  2. Modify your .csproj or .vbproj file, add <Import Project=”$(MSBuildProjectDirectory)\Precompile.targets” /> towards the end of the file.
  3. In your project’s configuration, set the “AspNetCompiler” property to “true” for whatever configurations you want (I have it enabled for the “Release” configuration in my case).
  4. If you also want to invoke aspnet_merge, you need to ensure the system that does the building has Visual Studio or the Windows SDK installed, then in your project’s configuration, set the “AspNetMerge” property to “true” for whatever configurations you want.

aspnet_merge is another interesting tool that merges assemblies. You can read about its benefits on the aspnet_merge MSDN page. I found that it is beneficial to use aspnet_merge almost always – the only problem I had was on one of my very large projects (with dozens of ASPX page and the total web site size being a few hundred megabytes), aspnet_merge took an exorbitantly long time. For example, on our normal project, it takes ~1 minute. On this very large project, it took over an hour.

Categories: Uncategorized Tags:

HTTP Response Caching for Java and Android

February 21st, 2012 15 comments

HTTP caching is both important, as it reduces bandwidth use and improves performance, and complex, as the rules are far from simple. In my experience, most Java and Android applications either don’t do HTTP caching, or they roll their own and up doing it wrong or in way too complicated a fashion. In other words, they create a non-standard, one off, unmaintainable solution. And IMHO, that’s no solution at all.

If you find yourself using HttpClient, you can use HttpClient-Cache, which is an easy drop in for Java. See my previous post about HttpClient-Cache for Android. But if you’re using HttpUrlConnection (aka java.net.URL.openConnection()), there’s no good solution for regular Java or Android. Well, in Android 4.0 and later, you can use HttpResponseCache, but with only a small percentage of Android devices using 4.0 or later, that’s not a terribly good solution. If you use Android 4.0+’s HttpResponseCache as recommended by Google, then all previous Android versions end up with no HTTP response caching – this causes excess load on your servers, slower performance for the app, and unnecessary bandwidth use.

To fix this problem, I grabbed all the code from AOSP that implements Android 4.0’s HttpResponseCache and made it a separate library. This library is easy to use, works on Java 1.5+, all versions of Android, and is licensed under APLv2 (just like everything else in AOSP). Really, there’s no reason not to use it! You can even use in Java server applications, such as those that use Spring.

To use it, if you’re using Maven, simply add this block to your pom.xml (all artifacts are in Maven Central):

If you’re not using Maven, you’ll need to add the httpresponsecache jar and its dependency, disklrucache.jar, to your project.

When your application starts, before it makes any HTTP requests, execute this method:
com.integralblue.httpresponsecache.HttpResponseCache.install(File directory, long maxSize);
If you’re using Android, and you want to use Android 4.0’s HttpResponseCache if it’s available, and fallback to this library if it’s not available:
final long httpCacheSize = 10 * 1024 * 1024; // 10 MiB
final File httpCacheDir = new File(getCacheDir(), "http");
try {
.getMethod("install", File.class, long.class)
.invoke(null, httpCacheDir, httpCacheSize);
} catch (Exception httpResponseCacheNotAvailable) {
Ln.d(httpResponseCacheNotAvailable, "android.net.http.HttpResponseCache not available, probably because we're running on a pre-ICS version   of Android. Using c$
com.integralblue.httpresponsecache.HttpResponseCache.install(httpCacheDir, httpCacheSize);
}catch(Exception e){
Ln.e(e, "Failed to set up com.integralblue.httpresponsecache.HttpResponseCache");

The source code to the library is available on GitHub. I’m already using it in my CallerID Android app. If you end up using this library, please leave me a comment.

That’s it – enjoy easy to use HTTP caching!

Categories: Uncategorized Tags:

jshint in msbuild

February 17th, 2012 2 comments

I recently had to add build time Javascript validation to an ASP.NET project. It took me quite a while to figure out how to do so in a (reasonably) maintainable, understandable way.

I’m using Visual Studio 2010, and the project targets .NET 3.5. The same approach would work fine if the project was targeting .NET 4.0.

I’m using NuGet to manage dependencies. The first thing I did was add node-jshint as a dependency of the project.

I opened the project’s file (something.csproj). I added a target:

<Target Name=”jshint”>
<JavaScript Include=”@(Content)” Condition=”%(Extension) == ‘.js'” />
<Exec Command=”&quot;$(node-jshint)\tools\jshint.bat&quot; &quot;%(JavaScript.FullPath)&quot; –reporter &quot;$(node-jshint)\tools\lib\vs_reporter.js&quot; –config &quot;$(MSBuildProjectDirectory)\jshintrc.json&quot;” ContinueOnError=”true” />

Make BeforeBuild depend on jshint:

<Target Name=”BeforeBuild” DependsOnTargets=”jshint”>

Add a new text file to the root of project called “jshintrc.json” If the file is included in the Visual Studio project, make sure the build action is “None” so Visual Studio doesn’t try to do anything with it. The file contents should look like this. The latest available version of node-json at this time, 0.5.5, doesn’t deal with a Byte Order Markers (BOM) in the jshintrc.json file, so when saving it, be sure the BOM isn’t included.

Now whenever Visual Studio builds the project, JSHint errors will appear in the VS error list just like all other types of errors. It runs JSHint on all .js files included in your project as content (the way .js should be included in your project).

Categories: Uncategorized Tags:

Best way to use HttpClient in Android

September 30th, 2011 4 comments

Many Android applications access the Internet resources over HTTP (and my projects are no exception). There are 2 common ways to do that: use Apache HttpClient 4.x (which is included in Android) or use HttpURLConnection (from Java). Google stated in a September 29, 2011 blog post that they prefer you use HttpURLConnection, but many apps and a large number of Java libraries already use HttpClient and won’t be changing soon (if ever). So HttpClient is here to stay.

With that in mind, the performance and footprint of HttpClient can vary widely based on how its set up. Here are my recommendations:

  • Always use one HttpClient instance for your entire application. HttpClient is not free to instantiate – each additional instance takes time to create and uses more memory. However, more importantly, using one instance allows HttpClient to pool and reuse connections along with other optimizations that can make big differences in how your application performs.
  • Use a thread safe connection manager. If you’re using one global HttpClient, it will be accessed by multiple threads concurrently – so if you don’t use a thread safe connection manager, Bad Things will happen.
  • Use Android’s android.net.SSLCertificateSocketFactory and android.net.SSLSessionCache if they’re available. Using these instead of the base HttpClient SSLSocketFactorywill reduce round trips when connecting to the same https site multiple times, making your application feel faster.
  • Set the user agent to something useful. That way, the server’s logs will be far more useful, which may save you (or someone else) a lot of time later if (when) a problem occurs.

With all that said, here’s how I get my global HttpClient instance. This code should work on all Android versions (it should even work all the way back to 1.0 – if anyone cares). I use Google Guice‘s Provider interface and injection to get the application, but you can easily adopt this to a form that doesn’t use Guice.

import java.lang.reflect.Method;
import java.util.Locale;
import org.apache.http.client.HttpClient;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.scheme.PlainSocketFactory;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.scheme.SocketFactory;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.impl.client.AbstractHttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.params.HttpParams;
import org.apache.http.params.HttpProtocolParams;
import android.app.Application;
import android.content.Context;
import android.content.pm.PackageManager.NameNotFoundException;
import android.os.Build;
import android.util.Log;
import com.google.inject.Inject;
import com.google.inject.Provider;
public class HttpClientProvider implements Provider {
Application application;
// Wait this many milliseconds max for the TCP connection to be established
private static final int CONNECTION_TIMEOUT = 60 * 1000;
// Wait this many milliseconds max for the server to send us data once the connection has been established
private static final int SO_TIMEOUT = 5 * 60 * 1000;
private String getUserAgent(String defaultHttpClientUserAgent){
String versionName;
try {
versionName = application.getPackageManager().getPackageInfo(
application.getPackageName(), 0).versionName;
} catch (NameNotFoundException e) {
throw new RuntimeException(e);
StringBuilder ret = new StringBuilder();
ret.append(" (");
ret.append("Linux; U; Android ");
ret.append("; ");
ret.append("; ");
ret.append(" ");
return ret.toString();
public HttpClient get() {
AbstractHttpClient client = new DefaultHttpClient(){
protected ClientConnectionManager createClientConnectionManager() {
SchemeRegistry registry = new SchemeRegistry();
new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));
new Scheme("https", getHttpsSocketFactory(), 443));
HttpParams params = getParams();
HttpConnectionParams.setConnectionTimeout(params, CONNECTION_TIMEOUT);
HttpConnectionParams.setSoTimeout(params, SO_TIMEOUT);
HttpProtocolParams.setUserAgent(params, getUserAgent(HttpProtocolParams.getUserAgent(params)));
return new ThreadSafeClientConnManager(params, registry);
/** Gets an HTTPS socket factory with SSL Session Caching if such support is available, otherwise falls back to a non-caching factory
* @return
protected SocketFactory getHttpsSocketFactory(){
try {
Class sslSessionCacheClass = Class.forName("android.net.SSLSessionCache");
Object sslSessionCache = sslSessionCacheClass.getConstructor(Context.class).newInstance(application);
Method getHttpSocketFactory = Class.forName("android.net.SSLCertificateSocketFactory").getMethod("getHttpSocketFactory", new Class[]{int.class, sslSessionCacheClass});
return (SocketFactory) getHttpSocketFactory.invoke(null, CONNECTION_TIMEOUT, sslSessionCache);
}catch(Exception e){
Log.e("HttpClientProvider", "Unable to use android.net.SSLCertificateSocketFactory to get a SSL session caching socket factory, falling back to a non-caching socket factory",e);
return SSLSocketFactory.getSocketFactory();
return client;

I use this approach in my CallerID app (source) and an upcoming app (that I cannot yet talk about). I’ve also submitted patches (which have been accepted) to reddit is fun, so it will use this approach in its next version.

Categories: Uncategorized Tags:

My First Android App: CallerID

April 18th, 2011 7 comments

I’ve been looking for an excuse to write an Android app, and those annoying “unknown number” phone calls presented themselves at the perfect problem to solve.
My CallerID application consists of two parts: a service that runs on a server and given a phone number returns the information associated with it, and an Android app that uses the service to display information to the user upon request or when the phone rings.

The service portion is licensed under the AGPLv3 so anyone can be automomous and run it themselves, instead of relying on my instance. I based the idea and some of the implementation, although not much of the original source is left, on CallerID Superfecta. It’s written in PHP, has pretty minimal requirements, and is easy to install – check out the source for the CallerID service on gitorious. In case you don’t want to install it yourself, you can use my instance, which the Android app uses by default. To query it, request URLs like this: http://callerid.integralblue.com/callerid.php?format=json&num=%28860%29%20429-7433 The service currently returns the listing name, company name (if applicable), and address in JSON format. The service can also just return the listing name, which is great for use with Asterisk, using URLs like this: http://callerid.integralblue.com/callerid.php?format=basic&num=%28860%29%20429-7433 To use this with Asterisk, make sure you have CURL support available, and modify your dialplan with a directive such as this one:
exten => _., n, Set(CALLERID(name)=${CURL(http://callerid.integralblue.com/callerid.php?format=basic&num=${CALLERID(num)})})

The Android application is written in Java, licensed under the GPLv3, and is available on the Google Android Market and (hopefully) soon on the Amazon Market and f-droid. The Android app source code is also available on gitorious. The application requires at least Android 1.5 – and supporting Android 1.5 all the way to 2.3 was an interesting experience, especially in the area of the Contacts API. It uses OpenStreetMaps (osmdroid) for maps (instead of Google Maps, as not every device has Google Maps, and Google Maps is not Free Software) and roboguice as an application framework.

I initially built the app using ant, like most Android apps, but decided to switch to using Maven for a few reasons:

  • Maven manages dependencies, so I don’t need to include large binaries in the source repository.
  • Using a Maven multimodule approach, I can have a separate integration testing project (application-it) and application project (application), making automatic testing easy.
  • I’ve used Maven for a number of projects in the past, and find it easy and sufficiently “magic” (but not overly so), compared to ant, which I find too “low level.”
  • Maven automatically will run proguard (for optimization, not obfuscation), zipalign, and signing when given a “release” profile, or skip those steps when running under the default profile.

I used the maven-android-plugin as a key component of the build process. If you’re starting an Android project, I highly recommend you check it out.

To build the app:

  1. Install the Android SDK (you’ll need at least SDK versions 3 and 10)
  2. Set the ANDROID_HOME environment variable as described on the maven-android-plugin Getting Started page
  3. Install Maven (Windows and Mac installers are available for download, and it’s packaged for most Linux distributions)
  4. Install Git (Windows and Mac installers are available for download, and it’s packaged for most Linux distributions)
  5. git clone git://gitorious.org/callerid-for-android/mainline.git mainline
  6. cd mainline/application
  7. mvn clean install
  8. Since osmdroid is not in the Maven repository, you’ll need to manually install it. These directions can also be found in application/pom.xml:
    1. wget http://osmdroid.googlecode.com/files/osmdroid-android-3.0.3.jar
    2. mvn install:install-file -DgroupId=org.osmdroid -DartifactId=osmdroid -Dversion=3.0.3 -Dpackaging=jar -Dfile=osmdroid-android-3.0.3.jar
  9. mvn clean install (again) – it should succeed

The apk is located at “mainline/application/target/callerid-1.0-SNAPSHOT.apk”

These steps do not perform the integration tests. If you want to run those, in the step above that says “cd mainline/application” run “cd mainline” instead. Note that you’ll either have to have your Android phone plugged in to your computer (and running in debug mode, recognized by adb) or an Android emulator running.

To create a ready-to-release apk, which includes running proguard, zipalign, and jarsign, run this command from the “mainline” directory: mvn clean install -Prelease -Dsign.keystore=/home/candrews/projects/callerid/test-key.keystore -Dsign.alias=mykey -Dsign.storepass=testtest -Dsign.keypass=testtest Note that this command will sign using the test key. If you really want to distribute the apk, you’ll need to generate your own key using keytool. In either case, the ready to distribute apk is located at “mainline/application/target/callerid-1.0-SNAPSHOT-signed-aligned.apk”

If you want to develop the application using Eclipse, first make sure that you can successfully compile the application using the steps above. Then you need to install some Eclipse plugins:

From Eclipse:

  1. Select File-Import…
  2. Maven, Existing Maven Projects
  3. The Root Directory is the “mainline” directory you checked out from git
  4. Finish
  5. You should now have 3 projects in your Eclipse workspace: callerid, callerid-it, and callerid-parent
  6. For the “callerid” and “callerid-it” project, modify the Build Path by right clicking on the project, selecting “Build Path” then “Configure Build Path.” You need to remove “JRE System Library” and then click “Add Library” “Android Classpath Container” (see https://code.google.com/a/eclipselabs.org/p/m2eclipse-android-integration/issues/detail?id=41)
  7. That’s it – Eclipse should automatically build as you go, and you can use the Eclipse/Android development tools just as everyone else does.

I hope both the server portion and application portion are reasonably easy to run, build, and understand. If you have any questions or concerns, please comment – and contributions are of course very welcome to both projects!
Please note that this project is not in any way related to my employer – it is a completely personal, non-work project. This article has been cross-posted to Isobar’s blog.

Categories: Uncategorized Tags: , , , ,

The Coming IPv6 Evolution

September 27th, 2010 No comments

The coming exhaustion of the IPv4 address space has been in the news for a few years now. Various organizations have warned that the end is nigh, and finally, it appears the transition to IPv6 is really starting to really pick up steam. The transition to IPv6 will provide more capability, more opportunity, more performance, and a generally better user (and administrator) experience.

The switch to IPv6 will impact every web site, every server, every device – every Internet user – over the next couple of years. If you haven’t already gotten started with your transition plan, now is the time to do so.

The Problem

IPv4 address are 32 bits, commonly expressed 4 octets in dotted quad notation, like this: x.x.x.x where x is a number between 0 and 254. People have been using and memorizing IP addresses for almost 30 years. For instance, your home network is probably using IP addresses in the range of 192.168.x.x.

Because the IPv4 addresses we all know and love are 32 bits, there are about 4 billion unique addresses. However, some of those addresses are reserved for special use and some for home networks. Also, in the 1980’s, before anyone realized just how big this Internet thing would be, addresses were assigned in a very inefficient manner: for example, MIT, Apple, Prudential Insurance, General Electric and IBM each have 8 million addresses – HP has 16 million. Naturally, because the Internet started in the US, all of these big block allocations were made to US entities. As other countries came online, and continue to come online, and as new technologies like virtualization, cell phone, tablets, TVs, PVRs, cars, even toasters join the Internet, the number of IP addresses available is simply insufficient. The consensus is that the last block of IP addresses will be allocated in 2012.


What happens after the last block of address is allocated? Probably not much – at least immediately. China and India would be affected first, as they have the largest online growth. ISPs will start charging more for service (or they won’t issue IPv4 addresses at all, issue only IPv6 addresses), as IP addresses which were formerly essentially free will now an increasingly high value. Markets will develop to buy and sell IP addresses. To avoid using addresses, ISPs will stop allocating Internet IP addresses to their customers, instead making up their own address space using NAT (like how your home router does this for your home network), essentially creating little Internets.

Over time, this “little Internets” problem will spread, and will occur throughout the world. There are 3 major issues with the “little Internets” “solution” to IPv4 address exhaustion:

  1. Devices on separate little Internets will not be able directly communicate – or at least not without jumping through very tricky hoops. Gaming, VoIP, and P2P all become less reliable, more difficult to develop, and harder to use.
  2. Hosting will become more expensive, crippling innovation. Right now, if you have the next big idea, you pay $10/month (maybe even less) for web hosting, $10/yr for a domain name, and you’re off. Imagine how crippling it would be if there was an additional IP address fee tacked on – perhaps starting at $5/month, then quickly jumping to much more than that.
  3. Because of damage done to P2P technologies and the added expense of hosting, the Internet will more quickly transform into less of a collaborative platform, and more into a consumer/provider platform, becoming more like Cable TV than the free for all open forum it is now.

IPv4 exhaustion will be very bad for the common person, even if he doesn’t understand the problem. However, it will be very good for current established institutions because it will become increasingly more expensive to compete with them. For example, Google started out as two college kids – they ended up taking down a billion dollar company’s product (DEC’s Altavista). With the added costs and challenges that would exist in a post-IPv4 exhaustion world, Google could not have possible become what it is today.

The Solution

The solution is to grow the address space – make more addresses. Sounds simple, right? Well, not really.

The size of the IPv4 address space is very firmly planted in the IPv4 protocol; there is no way to change it. As early as the late 1980’s, people began to realize that the address space was not big enough, so work on the next protocol began. Meanwhile, the world changed: mobile phones came about and become extremely popular, virtualization became popular (again), VoIP was invented, and computer science evolved new ideas.

In 1998, the specification for IPv6 was finalized. IPv6 includes optimizations for mobile devices (such as cell phones), quality of service is built in (so VoIP works better), security is built in, improved efficiency/performance, and the address space is vastly larger (128 bits instead of 32 bits). There are about 4 billion IPv4 address compared to the 3.4×1038 provided by IPv6. Considering as how there are 5×1028 for each human being alive today, IPv6 address space should last us a while.

Today, all modern operating systems (Windows XP, Linux 2.6, MacOS 10.3) support IPv6, and almost all software does.

Who’s Pushing IPv6

The US government realized that IPv4 exhaustion was going to be a problem, so it mandated that all new technology purchases be IPv6 capable, and all networks be upgraded to IPv6 by 2008. As with most government projects, the work is not 100% done, but many federal systems are running IPv6 right now.

China started the “China Next Generation Internet” project to push IPv6 adoption within its borders very hard. The 2008 Olympics websites all other network services were available over IPv6. China of courses realizes that if IPv6 isn’t adopted, and IPv4 doomsday comes along, it will be at a serious disadvantage.

ISPs also realized that without IPv6, service will become more expensive and cut into margin. So ISPs such as Comcast and Verizon are implementing IPv6.

In Australia and Europe, many ISPs have already implemented IPv6.

Mobile phone service providers are pushing IPv6 as mobile phones are major consumer of IP addresses. For this reason, T-Mobile is running an IPv6 test program, and Verizon is mandating that all new phones on its network be IPv6 capable in the near future.

Realizing that many of its customers will soon have IPv6 access, and may not always have IPv4 access, Google is moving its infrastructure to IPv6.

Other companies are realizing the same thing, such as Akamai and Facebook.

Should My Next Project Run On IPv6?

Yes. The benefits include:

  • Future proofing. IPv4 will be around for a long time still, but it won’t always be cheap, and it will become more complicated to administer. If your site will last more than a year, you should provide IPv6 connectivity.
  • Customer experience. If you serve only over IPv4, and a customer using only IPv6 visits your site, your site may not work (if it does work, it will be slower). If you serve only IPv4, and a customer supporting both IPv4 and IPv6 visits your site, the user may experience improved performance.

Supporting IPv6 can be easy. Ask your ISP or hosting provider if they support IPv6 (many backbone providers and large, commercial ISPs do) – if they don’t, go elsewhere. Your operating system already supports it, and chances are, the rest of your software does too.

As Cameron Byrne from T-Mobile USA repeatedly told content providers at Google’s IPv6 Implementor’s Conference:

Our users are going to access your content over IPv6. The only relevant question is “will we make the AAAA record or will you?” Wouldn’t you rather be the one to do it so you have control?

Cross posted to the Isobar blog – please comment there.

Categories: Uncategorized Tags: , ,