crawlajax

search engines never liked javascript and ajax. as a person who advocates seperation of HTML and data, SEO always became a problem for me in software design.

thanks google, finally they suggested a solution like this:

http://code.google.com/web/ajaxcrawling/docs/getting-started.html

if you use hashbang in your URL, then googlebot does a request to your backend with _escaped_fragment_ parameter in which your url fragment (the part after hashbang) is passed, so google can index the response you returned.

at backend you can render your frontend page via a tool using one of layout engines or native libraries generally used for html unit tests, then you can publish the DOM XML you gained for response.

google had suggested htmlunit for that purpose. but we had performance problems with htmlunit because our application was a very heavy javascript application and render time was too long.

at this point, for performance, i'll have 3 advice independent from tool you preferred:

• do not render CSS and be sure you're not doing requests for CSS files.
• do not request images, flash and etc.
• sign your requests done from your backend via a special user agent. by that way, you can understand it's googlebot request at javascript of your frontend and you can prevent unnecessary operations (like doing facebook and google analytics requests, these are really slowing rendering).

after performance problems, as an alternative, we wrote a crawler to create and save html snapshot of rendered DOM, and served created folder statically to googlebot.

for this, we used phantomjs for rendering, and wrote a small python program to crawl site:

https://github.com/ekesken/crawlajax

project satisfies only our application's requirements for now, you're free to fork and enrich it according to your needs.

when i googled with keywords 'ajax' and 'seo', there were many advices but there was no working application. i hope this project can fill this gap.

back to the kartaca

kartaca is a company founded under parkyeri. after my military service, i had worked for another company zeitin which is also founded under parkyeri. zeitin had taken money from parkyeri and kartaca had taken it's current customers and projects.

i had tried to summarize my transfer from zeitin to mynet in this post.

after a happy 1.5 year in mynet, i returned to kartaca with an uneasy decision. actually we can't say that it's a real return. because kartaca is a different company with it's new methodologies even if it'd taken over the jobs parkyeri does. it's a new organization settled on teams and goals.

i was satisfied with my job and i was happy with my colleagues in mynet. that made it hard to decide. i rationalized my decision referring to these reasons:

• income and enhancing income
  
• my aim to become a system engineer
  
• having a education budget
  
• giving/attending seminars is promoted
  
• being close to free software community
  
• having a library
  
• free chocolate
  
• being a technology company
  
• having a company ethic and vision
  

income and enhancing income: mynet couldn't provide same conditions that kartaca does. there is salary scale for programmer position in mynet, you can't get more than that limit. in software department there is an established personnel cadre for years, so there is no promotion hope either. as a result i had not much chance to increase my salary for near future. i thought i would have a more brilliant future in kartaca, and it was a right time to pass to a company which has a potential of growing.

my aim to become a system engineer: after 3 years i had chance to perform a decision i'd taken after my military service. i could also do this in mynet, but i had doubts about my happiness in system administration department. software department and system administration department people in mynet working in completely different conditions. i didn't do more than 3 overtimes in mynet during 1.5 year. but in system side people were always working overtime. reasons like that was preventing me from passing to system administration. by my transfer to kartaca, i had succeeded to pass to system engineering. i even determined my new title: "Site Reliability Engineer". i thing my new title would better serve my future goals. i don't want to do managing jobs and i want to stay in the kitchen of work. it's a pity that you can't increase your income by staying as a plain "software developer", you need to be a project or department manager or something like this. i can always continue programming as a hobby as well.

having a education budget: every employee in kartaca has piggy bank to save money for training purposes. they are putting 100$ to this moneybox every month. you can use this money for any education you prefer. actually i will collect several certificates in time just for taking advantage of this money. i wouldn't spend that money to education if they did give me it as cash. it's a good feeling that knowing your company thinks about your education and it's nice that they are leaving you free to decide which education you'll take.

giving/attending seminars is promoted: it's a company that promotes it's employees give seminars and attend seminars.

being close to free software community: companies founded under name parkyeri always supported free software community in turkey as monetary and morally. this gives employees chance to have a nice entourage.

having a library: i understand the value of this after losing it. i don't read a book twice, so it's meaningless for me to collect books. during my education i was using university library, after school i was taking advantage of parkyeri library. when i started to work in mynet, new concepts came into my life as giving money to books and having a library at home. you can order any book you want in kartaca no matter it's a technical book or not. also you can benefit from many foreign and local magazine subscriptions. it's amazing to find a technical book on shelf that you just searched on INTERNET. because it was already ordered by someone else. and non-technical books are important because they are turning reading activity into a social thing.

free chocolate: in mynet we were used to spend some amount of our salary to the regular "5 tea" meetings. i was always telling my friends about our free chocolate repository that we have in parkyeri. this is a nice and probably easiest way of making your employees happy with a little cost to company.

being a technology company: using technology and selling technology are totally different things. kartaca is a company that sells man-day, so what's commercialized and sold is the technical ability and experience of employees, nothing else. there is a win-win principle goes on. as long as i keep improving my technical skills, company's value is increasing as well. but mynet is a company surviving by member fees and ad revunue. as a technical guy, your part in company's success is not much in that case. it's easy for mynet to outsource all the technical jobs one day, but kartaca's existence reason is the technical manpower it has.

having a company ethic and vision: before i left parkyeri because of my military service, we had just done a work to determine vision/mision. kartaca had owned the result of that work and had taken serious steps in this way. company had managed to survive by fulfilling necessities of today's business, and as well kept proceeding through it's vision. while managing all these, never had sacrificed ethic values which are determined in this work. i can claim these for kartaca clearly: respecting and giving social rights of its' employees (even if in internship, you have a social security and salary, i don't believe that there is one more company in same scale with kartaca that does this in turkey), won't do black work to avoid taxes, won't use unlicensed software, won't do anything illegal at all. i wonder how many employee there exists in turkey, that can say same things without no doubt for his/her company.

java equivalent of php encryption with MCRYPT_3DES and MCRYPT_MODE_CBC

if you try to write a java code to produce same encrypted words which are generated with php using TripleDES algorithm and CBC mode, you'll see two main keywords at the end of your search on internet:

• DESede (as algorithm)
• DESede/CBC/PKCS5Padding (as transformation method)

key point here is transformation method, because php does not support "PKCS5Padding". It just adds empty bytes at the end of plain text to be encrypted to make it's length multiply of 8. so you must use "DESede/CBC/NoPadding" transformation method and do the same padding php does with hand not to get "invalid length"[1] errors.

you can see php code and it's java equivalent below.

PHP:

$plaintext = "Some-plain-text-message-to-be-symetrically-encrypted";
$deskey = "secret word with 24 byte";
$ivkey = "12345678";
$td = mcrypt_module_open(MCRYPT_3DES, '', MCRYPT_MODE_CBC, '');
mcrypt_generic_init($td, $deskey, $ivkey);
$encrypted_data = mcrypt_generic($td, $plaintext);
mcrypt_generic_deinit($td);
echo strtoupper(bin2hex($encrypted_data));
mcrypt_module_close($td);

JAVA

import javax.crypto.Cipher;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.DESedeKeySpec;
import javax.crypto.spec.IvParameterSpec;
import org.apache.commons.codec.binary.Hex;
import java.util.Date;
import java.util.Arrays;
import java.text.SimpleDateFormat;
import java.text.ParsePosition;

public class Encrypt {

public static void main(String[] args) throws Exception {
String plainText = "Some-plain-text-message-to-be-symetrically-encrypted";
String desKey = "secret word with 24 byte";
String ivKey = "12345678";
String algorithm = "DESede";
// String transformation = "DESede/CBC/PKCS5Padding";
String transformation = "DESede/CBC/NoPadding";
byte[] keyValue = desKey.getBytes("UTF-8");
byte[] ivValue = ivKey.getBytes("UTF-8");
DESedeKeySpec keySpec = new DESedeKeySpec(keyValue);
IvParameterSpec iv = new IvParameterSpec(ivValue);
SecretKey key = SecretKeyFactory.getInstance(algorithm).generateSecret(keySpec);
Cipher encrypter = Cipher.getInstance(transformation);
encrypter.init(Cipher.ENCRYPT_MODE, key, iv);
// byte[] input = plainText.getBytes();
byte[] input = getPaddedBytes(plainText);
byte[] encrypted = encrypter.doFinal(input);
System.out.println(new String(Hex.encodeHex(encrypted)).toUpperCase());
}

public static byte[] getPaddedBytes(String s) throws java.io.UnsupportedEncodingException {
int n = s.length();
n = n + (8 - (n % 8));
byte[] src = s.getBytes("UTF-8");
byte[] dst = Arrays.copyOf(src, n);
return src;
}
}

[1] javax.crypto.IllegalBlockSizeException: Input length not multiple of 8 bytes