Windows Development and Testing

This documents what you need to build and test on Windows

  1. Windows server 2012
  2. Create an admin account for yourself
  3. On a virtual machine: take snapshots as you go along

Core installs

  • Install openssl 1.0.1i 64-bit into a dir with no spaces, C:\bin\openssl. Add its bin dir to your path. (this will also need the Visual C++ 2008 Redistributables (DLL libraries) installed.

Virtual Box

  1. Ask for lots of RAM, CPU. You can vary CPU and limits as you go, and increase them for test runs/full screen work.
  2. 2D acceleration of video
  3. Critical: Enable PAE under "processor". this avoids random hangs.

OS features

In the server management app, go to "manage features", where you can add

  • Telnet server: lets you telnet in.
  • Maybe: desktop experience

OS settings

Firewall setup is a mystery. You need to convince the OS that your Ether network is "private", then edit the firewall to open it up.

Dev tools

  1. Bitvise SSH server; add your private key to the account you create. (even with this installed, you will still be mostly working from the windows command prompt on the machine, as it has the env variables set up)
  2. Git
  3. Java 7+
  4. Python 2.7
  5. OpenSSL
  6. Everything Apache Hadoop's BUILDING.TXT needs. This includes the cygwin tools, protoc, python and more. If you cannot build Hadoop from the Windows SDK Command Prompt then don't proceed.
  7. JEdit is a good text editor for developing and editing simple files.
  8. Chrome can be used a browser instead of IE if you want —but you should still view all web pages in IE to verify they look OK.
  9. Sysinternals. Their tcpview tool is ideal for seeing what ports Hadoop services are listening on.

Knowledge

  1. Knowledge of the CMD.EXE command line

  2. How to set Path

    1. In explorer, find "This PC"
    2. right mouse button onto "properties"
    3. then "advanced properties"
    4. then "Environment variables"
    5. then the user env vars ... scroll down to find path.
    6. copy the Path value, edit in jedit or similar, paste back in.
    7. Open a new command window.

Windows

  1. Follow the Hadoop BUILDING.TXT instructions.
  2. install cygwin64 with it's sh command.
  3. Get the cygwin64 bin dir on the PATH (remember, ";" as separators)
  4. find where the CL C/C++ compiler is and make sure it is on the PATH
  5. get the (native) CMake 2.8+ command on the path, not any cygwin one.

Things to install in cygwin sh ls rm cp gzip tar awk grep * xargs

Avoid

  • having cygwin cmake on your path; it must be the native cmake

env variables

CYGWIN=nodosfilewarning 
PATH = cmake bin, cygwin bin, python bin, hadoop_home bin (see below)

example:

C:\bin\cygwin64\bin;C:\bin\Python27\;C:\Windows\system32;C:\Windows;
C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;
c:\java\jdk7\bin;C:\Program Files\Microsoft Windows Performance Toolkit\;
C:\apps\maven\bin;C:\apps\Git\cmd;c:\bin;
C:\Program Files (x86)\MSBuild\12.0\Bin;
C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin;
C:\bin\cmake-2.8\bin;C:\bin\hadoop\bin;C:\bin\OpenSSL\bin

Commands to test CLI setup

cmake --version
openssl version
python --version
git --version
tar --version

java -version
mvn --version

msbuild /version
cl /help

and for hadoop, once installed

winutils sysinfo
hadoop version
hadoop checknative

Slider's test for windows support

You can test for windows support with the slider-core test

mvn test  -Dtest=TestWindowsSupport

This looks for the arguments and other valid settings

Building Hadoop

  1. Find the "Windows SDK 7.1 command prompt" using the search field in the windows 8 screen
  2. Run this, and "pin" it to the task bar. This will be the primary console for building hadoop.
  3. Right mouse button on the icon in the top-left corner to bring up the "defaults" dialog.
  4. Choose whatever colors and fonts you want
  5. In layout, select a wider window size than 80 (if you want)
  6. in "screen buffer", set the no. of lines to 999.
  7. close the window and re-open it to get the new settings

From a Windows SDK command prompt:

set HADOOP_VERSION=2.6.0

set HADOOP_VERSION=2.7.0-SNPSHOT

or

set HADOOP_VERSION=3.0.0-SNAPSHOT

in the root directory of the hadoop repository

mvn clean
mvn install package -Pdist -Pnative-win -Dtar -DskipTests -Dmaven.javadoc.skip=true

(use two commands as sometimes clean can fail with some recursive directory-in-use exception)

pushd hadoop-dist\target
tar -xvzf hadoop-%HADOOP_VERSION%.tar.gz
rm -rf c:\bin\hadoop
cp -r hadoop-%HADOOP_VERSION% c:\bin\hadoop
popd

Hadoop is now installed into c:\bin\%HADOOP_VERSION%

Set up the env variables by hand

 set HADOOP_HOME=C:\bin\hadoop
 set PATH=C:\bin\hadoop\bin;%PATH%

Verify the version is as expected

 hadoop version

Verify that the binaries are there

 hadoop checknative -a

This should report the hadoop binaries, even if it warns about missing compression libraries