Took me some time but I figured out how to do it using NuGet.
First I added a location on my PC as package source as explained here.
In VS2013 CE you do this by opening the NuGet Package Manager Settings (Tools > NuGet Package Manager > Package Manager Settings) and under Package Sources you add a location on your computer.
To create the NuGet I combined several HowTo's since one alone didn't work.
The main one was this.
I built the dlls for my required platforms and created the following folder structure

Since it might be a little hard to see ProjectName.props and ProjectName.targets are located in the netcore451 folder.
The .dll, .pri and .winmd in lib are the x86 version. According to the HowTo it's redundant and you can just ignore these files but without them VS might not work correctly in design mode.
In the folder that contains build and lib I created a file ProjectName.nuspec
<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/01/nuspec.xsd">
<metadata minClientVersion="2.5">
<id>ProjectName</id>
<version>1.0.0</version>
<authors>Stefan Fabian</authors>
<owners>Stefan Fabian</owners>
<requireLicenseAcceptance>false</requireLicenseAcceptance>
<description>Description</description>
<releaseNotes>First release.</releaseNotes>
<copyright>Copyright 2015</copyright>
<references>
<group targetFramework=".NETCore4.5.1">
<reference file="ProjectName.winmd" />
</group>
</references>
</metadata>
</package>
ProjectName.props
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
</Project>
ProjectName.targets
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Target Name="PlatformCheck" BeforeTargets="InjectReference"
Condition=" ( ('$(Platform)' != 'x86') AND ('$(Platform)' != 'AMD64') AND ('$(Platform)' != 'Win32') AND ('$(Platform)' != 'ARM') AND ('$(Platform)' != 'x64') )">
<Error Text="$(MSBuildThisFileName) does not work correctly on '$(Platform)'
platform. You need to specify platform (x86 / x64 or ARM)." />
</Target>
<Target Name="InjectReference" BeforeTargets="ResolveAssemblyReferences">
<ItemGroup Condition="'$(Platform)' == 'x86' or '$(Platform)' == 'Win32'">
<Reference Include="ProjectName">
<HintPath>$(MSBuildThisFileDirectory)x86\ProjectName.winmd</HintPath>
</Reference>
</ItemGroup>
<ItemGroup Condition="'$(Platform)' == 'x64' or '$(Platform)' == 'AMD64'">
<Reference Include="ProjectName">
<HintPath>$(MSBuildThisFileDirectory)x64\ProjectName.winmd</HintPath>
</Reference>
</ItemGroup>
<ItemGroup Condition="'$(Platform)' == 'ARM'">
<Reference Include="ProjectName">
<HintPath>$(MSBuildThisFileDirectory)ARM\ProjectName.winmd</HintPath>
</Reference>
</ItemGroup>
</Target>
</Project>
I'm not sure if it's necessary to check for x86 and Win32 but it works for me.
Disclaimer
This was the first time ever I created a NuGet package and the code above might suck.