我猜你已经猜对了，chalk 和 cheese 没有半毛钱关系，应该没有奶酪作的粉笔吧，as different as chalk and cheese 是一个习语，可以意为“完全不同的，天壤之别的”，它寓意：

for two things to be completely different from each other

两件事完全不同

to have nothing in common with someone or something else

与某人或某事没有共同点

to lack similarity when compared with someone or something

与某人或某事相比缺乏相似性

例如：

Despite James and John being identical twins, their characters were as different as chalk and cheese.

尽管詹姆斯和约翰是同卵双胞胎，但他们的性格却完全不同。

What he does in public and how in acts in private are as different as chalk and cheese.

他在公共场合所做的事和在私底下的行为是完全不同的。

Aside their names, the two teachers had nothing in common and could be said to be as different as chalk and cheese.

撇开他们的名字不谈，这两位老师没有什么共同点，可以说是天壤之别。

England in the Summer and England in the Winter are as different as chalk and cheese.

夏天的英国和冬天的英国是完全不同的。

所以这个习语也可以用来表达与某人没有共同语言，而且是一个比较高深的用法，即委婉，又能准确的表达意思​，例如：

We're as different as chalk and cheese.​

我和你没有共同语言​。